Oct 31 01:17:18.534517 ip-10-0-141-145 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 01:17:18.534529 ip-10-0-141-145 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 01:17:18.534536 ip-10-0-141-145 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 01:17:18.534753 ip-10-0-141-145 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 01:17:29.723030 ip-10-0-141-145 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Oct 31 01:17:29.723047 ip-10-0-141-145 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 45835c785ce94a048024a0d2f736b438 -- Oct 31 01:19:18.076010 ip-10-0-141-145 systemd[1]: Starting Kubernetes Kubelet... Oct 31 01:19:18.475941 ip-10-0-141-145 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 01:19:18.475941 ip-10-0-141-145 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 01:19:18.475941 ip-10-0-141-145 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 01:19:18.475941 ip-10-0-141-145 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 01:19:18.480868 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480785 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.480868 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480865 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.480868 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480869 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.480868 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480872 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.480868 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480875 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480879 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480881 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480884 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480887 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480890 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480893 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480896 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480898 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480901 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480904 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480908 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480911 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480913 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480916 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480919 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480921 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480924 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480926 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480929 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.481073 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480931 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480934 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480936 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480939 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480941 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480944 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480946 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480949 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480951 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480954 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480956 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480959 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480961 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480964 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480967 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480970 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480973 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480975 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480978 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480981 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.481528 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480984 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480987 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480990 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480992 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.480999 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481002 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481004 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481007 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481009 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481012 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481014 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481017 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481020 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481022 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481025 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481044 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481048 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481051 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481054 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481057 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.482014 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481059 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481062 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481065 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481068 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481071 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481073 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481077 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481080 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481082 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481085 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481088 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481090 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481093 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481096 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481098 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481101 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481105 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481109 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481112 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.482477 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481114 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481117 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481120 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481122 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481125 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481128 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481130 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481133 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481135 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481138 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481141 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481143 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481146 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481149 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481151 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481155 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481160 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481163 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481167 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.482944 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481171 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481176 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481179 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481182 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481954 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481962 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481965 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481968 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481970 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481973 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481976 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481978 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481982 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481984 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481987 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481990 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481993 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481995 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.481998 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.483417 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482001 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482003 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482007 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482011 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482014 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482018 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482021 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482023 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482026 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482029 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482032 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482034 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482038 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482041 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482043 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482046 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482049 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482051 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482054 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.483888 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482057 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482060 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482062 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482065 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482067 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482070 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482073 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482076 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482079 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482081 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482084 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482087 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482090 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482093 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482095 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482098 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482100 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482103 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482106 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.484344 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482109 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482111 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482114 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482117 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482119 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482122 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482125 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482127 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482130 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482132 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482135 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482138 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482140 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482143 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482145 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482148 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482151 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482153 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482156 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482158 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.484812 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482161 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482164 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482166 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482169 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482171 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482176 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482180 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482183 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482186 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482188 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482191 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482194 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482197 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482199 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482202 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482205 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482207 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482210 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482213 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482215 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.485273 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482218 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482220 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482223 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482225 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482228 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482231 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482233 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482235 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482238 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482240 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482243 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482245 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482248 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482764 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482770 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482773 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482776 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482779 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482781 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.485756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482784 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482787 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482790 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482792 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482795 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482798 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482800 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482803 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482806 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482809 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482811 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482814 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482817 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482820 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482823 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482826 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482829 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482831 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482834 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482837 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.486219 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482839 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482842 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482844 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482847 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482849 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482851 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482854 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482857 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482860 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482863 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482865 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482868 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482870 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482873 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482875 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482878 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482883 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482886 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482888 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.486688 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482891 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482894 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482896 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482899 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482902 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482904 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482907 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482910 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482912 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482915 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482917 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482920 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482923 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482925 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482928 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482930 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482933 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482936 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482938 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482941 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.487143 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482943 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482946 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482949 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482952 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482954 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482957 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482960 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482962 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482965 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482969 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482971 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482976 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482980 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482983 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482986 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482989 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482992 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482994 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.482997 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.487648 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483001 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483004 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483007 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483010 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483015 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483017 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483020 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483022 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483025 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483027 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483030 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483032 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483035 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483038 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483041 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483044 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483047 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483049 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483052 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483054 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.488213 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483057 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.483060 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483941 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483951 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483957 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483962 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483967 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483970 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483978 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483982 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483985 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483988 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483992 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483996 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.483999 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484002 2575 flags.go:64] FLAG: --cgroup-root="" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484007 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484010 2575 flags.go:64] FLAG: --client-ca-file="" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484013 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484016 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484021 2575 flags.go:64] FLAG: --cluster-domain="" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484024 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484027 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484031 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 01:19:18.488697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484034 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484038 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484041 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484044 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484048 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484052 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484055 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484058 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484061 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484064 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484068 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484071 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484074 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484077 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484080 2575 flags.go:64] FLAG: --enable-server="true" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484083 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484087 2575 flags.go:64] FLAG: --event-burst="100" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484090 2575 flags.go:64] FLAG: --event-qps="50" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484093 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484096 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484099 2575 flags.go:64] FLAG: --eviction-hard="" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484103 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484106 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484109 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484114 2575 flags.go:64] FLAG: --eviction-soft="" Oct 31 01:19:18.489305 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484117 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484120 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484123 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484126 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484129 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484133 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484135 2575 flags.go:64] FLAG: --feature-gates="" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484140 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484143 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484146 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484149 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484153 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484157 2575 flags.go:64] FLAG: --help="false" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484160 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484163 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484166 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484169 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484173 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484176 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484179 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484182 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484185 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484188 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 01:19:18.490527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484191 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484195 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484197 2575 flags.go:64] FLAG: --kube-reserved="" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484200 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484203 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484206 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484209 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484212 2575 flags.go:64] FLAG: --lock-file="" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484217 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484220 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484223 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484228 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484231 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484235 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484238 2575 flags.go:64] FLAG: --logging-format="text" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484241 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484244 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484247 2575 flags.go:64] FLAG: --manifest-url="" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484250 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484255 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484258 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484262 2575 flags.go:64] FLAG: --max-pods="110" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484265 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484269 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484272 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 01:19:18.491198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484277 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484280 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484282 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484286 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484293 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484296 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484299 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484303 2575 flags.go:64] FLAG: --pod-cidr="" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484306 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484308 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484312 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484315 2575 flags.go:64] FLAG: --port="10250" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484318 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484321 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-085afaf3362abe062" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484324 2575 flags.go:64] FLAG: --qos-reserved="" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484329 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484332 2575 flags.go:64] FLAG: --register-node="true" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484335 2575 flags.go:64] FLAG: --register-with-taints="" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484339 2575 flags.go:64] FLAG: --registry-burst="10" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484342 2575 flags.go:64] FLAG: --registry-qps="5" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484345 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484348 2575 flags.go:64] FLAG: --reserved-memory="" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484352 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484355 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484358 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484361 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 01:19:18.491964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484364 2575 flags.go:64] FLAG: --runonce="false" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484367 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484370 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484373 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484380 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484383 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484387 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484391 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484394 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484397 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484400 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484403 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484405 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484409 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484412 2575 flags.go:64] FLAG: --system-cgroups="" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484415 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484420 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484423 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484426 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484430 2575 flags.go:64] FLAG: --tls-min-version="" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484433 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484437 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484440 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484443 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484446 2575 flags.go:64] FLAG: --v="2" Oct 31 01:19:18.492968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484451 2575 flags.go:64] FLAG: --version="false" Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484455 2575 flags.go:64] FLAG: --vmodule="" Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484459 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484462 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484598 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484601 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484604 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484607 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484609 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484612 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484615 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484618 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484621 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484626 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484629 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484631 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484634 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484636 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484639 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484641 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.493590 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484644 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484647 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484649 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484652 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484654 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484658 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484662 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484665 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484669 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484672 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484675 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484677 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484680 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484682 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484685 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484688 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484690 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484693 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484696 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.494375 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484698 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484701 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484704 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484706 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484709 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484727 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484734 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484737 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484740 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484743 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484746 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484748 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484751 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484753 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484756 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484759 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484761 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484764 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484767 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484769 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.494871 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484772 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484776 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484779 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484781 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484784 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484787 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484789 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484792 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484794 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484797 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484799 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484802 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484806 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484810 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484814 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484818 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484821 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484824 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484828 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.495342 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484831 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484834 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484837 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484839 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484842 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484845 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484847 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484850 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484853 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484855 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484858 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484860 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484862 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484865 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484869 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484871 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484874 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484876 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484879 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.495808 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484881 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484884 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484886 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484889 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484891 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484894 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484896 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484899 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484901 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484904 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484906 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484909 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484914 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.484941 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484975 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.496264 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484980 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484983 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484986 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484989 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484992 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484995 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.484998 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485000 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485003 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485006 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485009 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485012 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485014 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485017 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485019 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485022 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485025 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485027 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485030 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.496605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485032 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485035 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485037 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485040 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485042 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485045 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485048 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485050 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485053 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485055 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485058 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485062 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485065 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485068 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485071 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485073 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485076 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485078 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485081 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485083 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.497078 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485086 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485088 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485091 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485094 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485099 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485102 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485105 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485107 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485110 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485112 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485115 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485117 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485120 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485122 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485125 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485128 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485131 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485133 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485136 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.497548 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485139 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485141 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485144 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485147 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485150 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485153 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485155 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485159 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485161 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485164 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485166 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485169 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485171 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485174 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485176 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485179 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485182 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485184 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485187 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485190 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.498024 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485192 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485194 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485197 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485199 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485202 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485205 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485207 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485210 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485212 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485215 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485219 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485222 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485226 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485228 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485231 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485233 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485237 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485239 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485242 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.498494 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485245 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485248 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485251 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485253 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485256 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485259 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485262 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.485264 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.492911 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.492932 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.493807 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.493817 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494143 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494154 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494158 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494162 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494165 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494167 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494170 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.498999 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494173 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494176 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494180 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494185 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494189 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494193 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494197 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494201 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494205 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494208 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494211 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494214 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494217 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494220 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494222 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494225 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494228 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494231 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494233 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.499506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494236 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494239 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494242 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494245 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494248 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494251 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494254 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494257 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494259 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494262 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494265 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494267 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494270 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494273 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494275 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494278 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494281 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494284 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494286 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494289 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.500038 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494292 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494294 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494297 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494299 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494302 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494305 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494307 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494310 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494312 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494315 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494317 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494320 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494323 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494325 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494329 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494332 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494334 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494337 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494339 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494342 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.500506 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494345 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494347 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494350 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494354 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494357 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494359 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494361 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494365 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494368 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494371 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494373 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494376 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494379 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494383 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494386 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494389 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494391 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494394 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494396 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.501027 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494399 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494401 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494404 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494406 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494409 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494411 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494414 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494416 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494420 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494422 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494425 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494427 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494430 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494432 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494435 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494437 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494440 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494442 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494445 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494447 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.501475 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494450 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.494478 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494629 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494634 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494638 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494641 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494645 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494648 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494651 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494654 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494656 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494659 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494661 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494664 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494667 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.501965 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494670 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494672 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494675 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494678 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494680 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494683 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494685 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494688 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494690 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494693 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494696 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494698 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494701 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494703 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494706 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494708 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494847 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494852 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494854 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.502322 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494857 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494860 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494863 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494866 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494868 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494870 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494875 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494877 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494880 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494883 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494886 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494889 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494892 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494895 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494897 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494900 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494902 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494905 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494908 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494910 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.502837 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494913 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494915 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494918 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494920 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494924 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494929 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494932 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494935 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494938 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494940 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494943 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494945 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494948 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494950 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494953 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494955 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494958 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494960 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494963 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494966 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.503312 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494968 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494971 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494974 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494977 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494979 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494982 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494985 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494987 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494990 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494992 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494995 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.494997 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495001 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495005 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495008 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495011 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495014 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495017 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495019 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.503876 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495022 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495025 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495027 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495030 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495032 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495035 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495037 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495040 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495043 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495045 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495048 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495051 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495053 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495056 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495059 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.495083 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 01:19:18.504341 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495114 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495118 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495121 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495124 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495127 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495129 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495132 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495135 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495137 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495140 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495142 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495145 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495147 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495150 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495152 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495155 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495158 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495160 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495162 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:18.504740 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495165 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495168 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495170 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495173 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495175 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495178 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495180 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495183 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495185 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495189 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495193 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495196 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495199 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495202 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495205 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495207 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495210 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495213 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495215 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495218 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:18.505182 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495220 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495223 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495225 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495227 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495230 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495240 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495243 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495246 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495249 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495251 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495254 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495257 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495259 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495261 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495264 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495266 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495269 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495273 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495276 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495279 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:18.505660 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495282 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495284 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495287 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495290 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495293 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495295 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495298 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495301 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495303 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495306 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495308 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495311 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495313 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495316 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495319 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495321 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495324 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495326 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495329 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495332 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:18.506281 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495334 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495337 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495340 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495343 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495345 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495348 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495350 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495353 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495356 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495358 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495361 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495363 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495366 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495368 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495371 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495373 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495376 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495380 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495383 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:18.506756 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495385 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495387 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495390 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495394 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495396 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495399 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495401 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:18.495404 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.495457 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.499700 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.500560 2575 server.go:1014] "Starting client certificate rotation" Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.500655 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.500691 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.502605 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 01:19:18.507204 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.503670 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 01:19:18.514997 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.514983 2575 log.go:25] "Validated CRI v1 runtime API" Oct 31 01:19:18.518058 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.518045 2575 log.go:25] "Validated CRI v1 image API" Oct 31 01:19:18.518934 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.518893 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 01:19:18.521266 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.521249 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cafca5fa-8b32-4e73-a54c-6a5b37301dd3:/dev/nvme0n1p3 cda684f1-2021-41db-b7f8-33cf0b76e508:/dev/nvme0n1p4] Oct 31 01:19:18.521313 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.521266 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 01:19:18.526785 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.526658 2575 manager.go:217] Machine: {Timestamp:2026-10-31 01:19:18.525298963 +0000 UTC m=+0.336227866 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088080 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c567a0b7fa0c8e09ad7a0a141bb79 SystemUUID:ec2c567a-0b7f-a0c8-e09a-d7a0a141bb79 BootID:45835c78-5ce9-4a04-8024-a0d2f736b438 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:86:28:f9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:86:28:f9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:85:19:50:36:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 01:19:18.526785 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.526784 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 01:19:18.526908 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.526854 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 01:19:18.529463 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.529435 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 01:19:18.529580 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.529465 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-145.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 01:19:18.529628 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.529586 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 01:19:18.529628 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.529594 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 01:19:18.529628 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.529626 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 01:19:18.529704 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.529640 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 01:19:18.530408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.530398 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 01:19:18.533017 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.533003 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 01:19:18.533126 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.533117 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 01:19:18.533870 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.533861 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 31 01:19:18.533899 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.533874 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 01:19:18.533899 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.533884 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 01:19:18.533899 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.533892 2575 kubelet.go:405] "Adding apiserver pod source" Oct 31 01:19:18.534020 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.533900 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 01:19:18.534229 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.534212 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 01:19:18.536815 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.536786 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 01:19:18.540412 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.540390 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 01:19:18.541949 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.541936 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 01:19:18.542670 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542658 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 01:19:18.542707 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542688 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 01:19:18.542852 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542842 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 01:19:18.542885 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542858 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 01:19:18.542885 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542865 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 01:19:18.542885 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542870 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 01:19:18.542885 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542876 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 01:19:18.542885 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.542881 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 01:19:18.543441 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.543431 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 01:19:18.543475 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.543442 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 01:19:18.543475 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.543450 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 01:19:18.543475 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.543456 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 01:19:18.543475 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.543464 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 01:19:18.543475 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.543476 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 01:19:18.544028 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.544016 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 01:19:18.544066 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.544030 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 01:19:18.547971 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.547957 2575 server.go:1263] "Started kubelet" Oct 31 01:19:18.548126 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.548075 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 01:19:18.548175 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.548155 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 01:19:18.548471 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.548445 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 01:19:18.548698 ip-10-0-141-145 systemd[1]: Started Kubernetes Kubelet. Oct 31 01:19:18.548961 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.548942 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-145.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 01:19:18.549350 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.549241 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 01:19:18.550393 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.550378 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 31 01:19:18.550643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.550623 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 01:19:18.550787 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.550770 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-145.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 01:19:18.553747 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.553728 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 01:19:18.555130 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.554898 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 01:19:18.556833 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.556803 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:18.557325 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.557308 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 01:19:18.557472 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.557366 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 01:19:18.557538 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.557476 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 01:19:18.557592 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.557559 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 31 01:19:18.557592 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.557568 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 31 01:19:18.557961 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.557947 2575 factory.go:55] Registering systemd factory Oct 31 01:19:18.558028 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.557965 2575 factory.go:223] Registration of the systemd container factory successfully Oct 31 01:19:18.558273 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.558227 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 01:19:18.558336 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.558278 2575 factory.go:153] Registering CRI-O factory Oct 31 01:19:18.558336 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.558291 2575 factory.go:223] Registration of the crio container factory successfully Oct 31 01:19:18.558423 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.558341 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 01:19:18.558423 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.558368 2575 factory.go:103] Registering Raw factory Oct 31 01:19:18.558423 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.558382 2575 manager.go:1199] Started watching for new ooms in manager Oct 31 01:19:18.559424 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.559410 2575 manager.go:319] Starting recovery of all containers Oct 31 01:19:18.564312 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.564281 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-145.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 01:19:18.567623 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.564143 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-145.us-east-2.compute.internal.18e3787dda0a0d34 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-145.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-145.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:18.547926324 +0000 UTC m=+0.358855230,LastTimestamp:2026-10-31 01:19:18.547926324 +0000 UTC m=+0.358855230,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-145.us-east-2.compute.internal,}" Oct 31 01:19:18.572158 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.572133 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 01:19:18.573050 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.572995 2575 manager.go:324] Recovery completed Oct 31 01:19:18.576694 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.576679 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:18.578306 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.578287 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:18.578379 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.578321 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:18.578379 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.578334 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:18.578893 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.578878 2575 cpu_manager.go:226] "Starting" policy="none" Oct 31 01:19:18.578893 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.578893 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 01:19:18.578986 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.578910 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 01:19:18.578986 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.578885 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:18.579646 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.579631 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:18.579737 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.579663 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:18.579737 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.579685 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:18.579987 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.579976 2575 policy_none.go:50] "Start" Oct 31 01:19:18.580024 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.579992 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 01:19:18.580024 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.580001 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 01:19:18.580870 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.580860 2575 policy_none.go:44] "Start" Oct 31 01:19:18.581226 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.581154 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-145.us-east-2.compute.internal.18e3787ddbd99b42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-145.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-145.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-145.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:18.578305858 +0000 UTC m=+0.389234762,LastTimestamp:2026-10-31 01:19:18.578305858 +0000 UTC m=+0.389234762,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-145.us-east-2.compute.internal,}" Oct 31 01:19:18.591819 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.591760 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-145.us-east-2.compute.internal.18e3787ddbd9f2e7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-145.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-145.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-145.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:18.578328295 +0000 UTC m=+0.389257199,LastTimestamp:2026-10-31 01:19:18.578328295 +0000 UTC m=+0.389257199,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-145.us-east-2.compute.internal,}" Oct 31 01:19:18.597246 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.597192 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-145.us-east-2.compute.internal.18e3787ddbda1af7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-145.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-145.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-145.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:18.578338551 +0000 UTC m=+0.389267456,LastTimestamp:2026-10-31 01:19:18.578338551 +0000 UTC m=+0.389267456,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-145.us-east-2.compute.internal,}" Oct 31 01:19:18.609401 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.609256 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrp7g" Oct 31 01:19:18.612743 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.612669 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-145.us-east-2.compute.internal.18e3787ddbd99b42\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-145.us-east-2.compute.internal.18e3787ddbd99b42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-145.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-145.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-145.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:18.578305858 +0000 UTC m=+0.389234762,LastTimestamp:2026-10-31 01:19:18.579646878 +0000 UTC m=+0.390575790,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-145.us-east-2.compute.internal,}" Oct 31 01:19:18.619776 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.619759 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jrp7g" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.622528 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.627706 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-145.us-east-2.compute.internal.18e3787ddbd9f2e7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-145.us-east-2.compute.internal.18e3787ddbd9f2e7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-145.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-145.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-145.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 01:19:18.578328295 +0000 UTC m=+0.389257199,LastTimestamp:2026-10-31 01:19:18.579679021 +0000 UTC m=+0.390607924,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-145.us-east-2.compute.internal,}" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.629663 2575 manager.go:341] "Starting Device Plugin manager" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.629733 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.629749 2575 server.go:87] "Starting device plugin registration server" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.629978 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.629987 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.630064 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.630128 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.630136 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.630658 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.630692 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:18.640097 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.639124 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 01:19:18.641733 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.640796 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 01:19:18.641733 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.640815 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 01:19:18.641733 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.640833 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 01:19:18.641733 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.640869 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 01:19:18.648829 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.648811 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:18.730778 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.730726 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:18.731463 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.731438 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:18.731554 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.731468 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:18.731554 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.731482 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:18.731554 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.731509 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.741567 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.741544 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal"] Oct 31 01:19:18.741636 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.741605 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:18.742136 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.742123 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.742206 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.742146 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-145.us-east-2.compute.internal\": node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:18.742343 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.742328 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:18.742407 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.742358 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:18.742407 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.742370 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:18.743563 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.743551 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:18.743693 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.743678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.743777 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.743755 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:18.744265 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.744250 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:18.744314 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.744272 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:18.744314 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.744283 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:18.744397 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.744313 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:18.744397 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.744330 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:18.744397 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.744339 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:18.745299 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.745286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.745340 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.745308 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:18.745970 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.745955 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:18.746049 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.745989 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:18.746049 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.746000 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:18.758584 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.758567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.771081 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.771063 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" node="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.773917 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.773904 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:18.774729 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.774703 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" node="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.859183 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.859165 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.859280 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.859192 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.859280 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.859209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.859280 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.859247 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.874369 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.874349 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:18.960233 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.960213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.960320 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.960236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.960320 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.960261 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.960320 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:18.960300 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:18.975342 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:18.975324 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:19.074056 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.074027 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:19.075844 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.075830 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:19.078452 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.078441 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:19.175940 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.175914 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:19.276444 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.276418 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:19.376957 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.376894 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:19.477497 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.477470 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:19.500641 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.500619 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 01:19:19.500787 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.500764 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 01:19:19.533160 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.533142 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:19.534158 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.534144 2575 apiserver.go:52] "Watching apiserver" Oct 31 01:19:19.544530 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.544514 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 01:19:19.544808 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.544792 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-pmc4f","openshift-multus/multus-additional-cni-plugins-h7k55","openshift-multus/multus-xfqs2","openshift-network-diagnostics/network-check-target-klvct","kube-system/global-pull-secret-syncer-c5d98","kube-system/konnectivity-agent-7rq4g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps","openshift-cluster-node-tuning-operator/tuned-bf87z","openshift-network-operator/iptables-alerter-fcqc8","openshift-multus/network-metrics-daemon-wff52","openshift-ovn-kubernetes/ovnkube-node-kjrt4","openshift-dns/node-resolver-mz75q"] Oct 31 01:19:19.546163 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.546145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.547217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.547200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.548084 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.548061 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.548945 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.548928 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.549993 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.549980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.550918 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.550904 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:19.551009 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.550970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:19.551242 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551214 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 01:19:19.551353 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551289 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vj55m\"" Oct 31 01:19:19.551353 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551306 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 01:19:19.551353 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551327 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 01:19:19.551521 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551378 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 01:19:19.551521 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551444 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnh8f\"" Oct 31 01:19:19.551521 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551485 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 01:19:19.551730 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551527 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 01:19:19.551730 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551552 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 01:19:19.551976 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.551957 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.553024 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553010 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.553516 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553469 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 01:19:19.553586 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553549 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 01:19:19.553586 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553564 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjbgx\"" Oct 31 01:19:19.553736 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553694 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 01:19:19.553736 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553730 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 01:19:19.553887 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553777 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82mbc\"" Oct 31 01:19:19.553887 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553783 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 01:19:19.553887 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.553802 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 01:19:19.554079 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.554067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:19.554133 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.554106 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:19.554250 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.554238 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:19.554316 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.554303 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 01:19:19.555309 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.555292 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.555370 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.555354 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:19.555756 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.555743 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gv7wl\"" Oct 31 01:19:19.556417 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.556403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.557079 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.557064 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:19.557658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.557640 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.558457 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.558443 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 01:19:19.559410 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.559395 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbmr\"" Oct 31 01:19:19.563464 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-dbus\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.563541 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-system-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.563541 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-daemon-config\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.563541 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-multus-certs\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.563541 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563517 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-etc-kubernetes\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.563680 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563559 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-sys\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.563680 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-node-log\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.563680 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-env-overrides\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.563680 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29151040-d2de-4a3c-b608-dc74c6aeee0e-serviceca\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.563825 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.563825 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cni-binary-copy\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.563825 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563763 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-netns\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.563825 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-device-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.563825 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-var-lib-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.563973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e137e152-115f-4459-80fe-a92048049da2-konnectivity-ca\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.563973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-host-slash\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.563973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563871 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29151040-d2de-4a3c-b608-dc74c6aeee0e-host\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.563973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-registration-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.563973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-bin\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.563973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnjx7\" (UniqueName: \"kubernetes.io/projected/29151040-d2de-4a3c-b608-dc74c6aeee0e-kube-api-access-rnjx7\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.563973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-kubernetes\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.563979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovn-node-metrics-cert\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e137e152-115f-4459-80fe-a92048049da2-agent-certs\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-hostroot\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-sys-fs\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmsj\" (UniqueName: \"kubernetes.io/projected/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kube-api-access-9dmsj\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.564236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-config\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564273 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97bgp\" (UniqueName: \"kubernetes.io/projected/930232e3-20a2-4134-99ab-9817dede3186-kube-api-access-97bgp\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-socket-dir-parent\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564337 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lxl\" (UniqueName: \"kubernetes.io/projected/5cf30068-f986-4286-b700-f68bdc0e5f75-kube-api-access-q5lxl\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-ovn\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-netd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-modprobe-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-run\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564537 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66jj\" (UniqueName: \"kubernetes.io/projected/4af660b2-97ac-4db7-b928-9d660b0c15a9-kube-api-access-z66jj\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-systemd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-lib-modules\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.564658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-host\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdsp\" (UniqueName: \"kubernetes.io/projected/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-kube-api-access-4zdsp\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkc6\" (UniqueName: \"kubernetes.io/projected/72564aed-4e32-4815-8a3e-6f61a8c36e83-kube-api-access-lhkc6\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-kubelet-config\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-socket-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-iptables-alerter-script\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-kubelet\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-conf\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-systemd\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.564972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-var-lib-kubelet\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-slash\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-etc-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphc2\" (UniqueName: \"kubernetes.io/projected/43b11ef2-886e-4d57-9700-089f51fa360e-kube-api-access-xphc2\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzdq\" (UniqueName: \"kubernetes.io/projected/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-kube-api-access-fdzdq\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.565135 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cnibin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-systemd-units\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-log-socket\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-script-lib\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565293 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-bin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565338 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/930232e3-20a2-4134-99ab-9817dede3186-hosts-file\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930232e3-20a2-4134-99ab-9817dede3186-tmp-dir\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565384 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-k8s-cni-cncf-io\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565398 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-tuned\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565448 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysconfig\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-tmp\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-system-cni-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.565760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-cnibin\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.566178 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-kubelet\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.566178 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-netns\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.566178 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.566178 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-multus\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.566178 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-conf-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.566178 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.565761 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.568351 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.568338 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 01:19:19.568390 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.568382 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal"] Oct 31 01:19:19.568423 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.568387 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:19.571660 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.571644 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 01:19:19.583098 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.583083 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 01:19:19.583179 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.583113 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal"] Oct 31 01:19:19.600230 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.600212 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:19.628596 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.628505 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 01:14:18 +0000 UTC" deadline="2028-05-30 04:37:07.095250261 +0000 UTC" Oct 31 01:19:19.628596 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.628552 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13851h17m47.466701213s" Oct 31 01:19:19.642596 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.642559 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j7g55" Oct 31 01:19:19.651491 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651451 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 01:19:19.651599 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651586 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 01:19:19.651664 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651596 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 01:19:19.651664 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651616 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 01:19:19.651664 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651647 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:19.651664 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651661 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 01:19:19.651902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651677 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 01:19:19.651902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651699 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 01:19:19.651902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651816 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h7598\"" Oct 31 01:19:19.651902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651829 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 01:19:19.651902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651855 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm2lp\"" Oct 31 01:19:19.651902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.651869 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 01:19:19.653914 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.653899 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 01:19:19.653995 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.653956 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 01:19:19.654051 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.653996 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5hw4\"" Oct 31 01:19:19.654051 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.654029 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 01:19:19.654835 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.654822 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j7g55" Oct 31 01:19:19.666412 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:19.666506 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-modprobe-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666506 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-run\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666506 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66jj\" (UniqueName: \"kubernetes.io/projected/4af660b2-97ac-4db7-b928-9d660b0c15a9-kube-api-access-z66jj\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.666506 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-systemd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.666506 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-lib-modules\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-systemd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666579 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-lib-modules\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666558 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-modprobe-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-host\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-run\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdsp\" (UniqueName: \"kubernetes.io/projected/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-kube-api-access-4zdsp\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.666566 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkc6\" (UniqueName: \"kubernetes.io/projected/72564aed-4e32-4815-8a3e-6f61a8c36e83-kube-api-access-lhkc6\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666659 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-host\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.666732 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs podName:43b11ef2-886e-4d57-9700-089f51fa360e nodeName:}" failed. No retries permitted until 2026-10-31 01:19:20.166700326 +0000 UTC m=+1.977629234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs") pod "network-metrics-daemon-wff52" (UID: "43b11ef2-886e-4d57-9700-089f51fa360e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-kubelet-config\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.666794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-socket-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-iptables-alerter-script\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666828 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-kubelet-config\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-kubelet\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-conf\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666896 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-kubelet\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666909 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-systemd\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-var-lib-kubelet\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-socket-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-slash\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666967 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-systemd\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667003 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-var-lib-kubelet\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667009 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-conf\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666985 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667014 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-etc-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.666978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-etc-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.667383 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667028 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-slash\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphc2\" (UniqueName: \"kubernetes.io/projected/43b11ef2-886e-4d57-9700-089f51fa360e-kube-api-access-xphc2\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667067 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzdq\" (UniqueName: \"kubernetes.io/projected/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-kube-api-access-fdzdq\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cnibin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667108 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-systemd-units\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-log-socket\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667156 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cnibin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667174 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-script-lib\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667184 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-systemd-units\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-bin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667206 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/930232e3-20a2-4134-99ab-9817dede3186-hosts-file\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667241 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-bin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930232e3-20a2-4134-99ab-9817dede3186-tmp-dir\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-k8s-cni-cncf-io\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667204 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-log-socket\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667298 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/930232e3-20a2-4134-99ab-9817dede3186-hosts-file\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.668300 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-tuned\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667365 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-k8s-cni-cncf-io\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667371 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysconfig\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-tmp\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667463 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-system-cni-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667472 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-iptables-alerter-script\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-cnibin\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-kubelet\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930232e3-20a2-4134-99ab-9817dede3186-tmp-dir\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-netns\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667619 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667631 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667655 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-cnibin\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.669208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667672 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-system-cni-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-kubelet\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667687 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysconfig\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-multus\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667748 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-netns\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-conf-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-dbus\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667843 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-script-lib\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667850 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-system-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667900 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667908 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-conf-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667911 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-system-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-multus\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.667941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-daemon-config\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-multus-certs\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.669699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-etc-kubernetes\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668087 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-dbus\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-sys\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-node-log\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-env-overrides\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668153 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-sys\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29151040-d2de-4a3c-b608-dc74c6aeee0e-serviceca\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668217 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-etc-kubernetes\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668318 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cni-binary-copy\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668328 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-netns\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-device-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-var-lib-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668440 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-daemon-config\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668474 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-env-overrides\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668484 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-netns\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.670681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e137e152-115f-4459-80fe-a92048049da2-konnectivity-ca\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-host-slash\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29151040-d2de-4a3c-b608-dc74c6aeee0e-host\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-registration-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-bin\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnjx7\" (UniqueName: \"kubernetes.io/projected/29151040-d2de-4a3c-b608-dc74c6aeee0e-kube-api-access-rnjx7\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668746 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-kubernetes\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668830 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-kubernetes\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668838 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-host-slash\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668871 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovn-node-metrics-cert\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668875 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-multus-certs\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e137e152-115f-4459-80fe-a92048049da2-agent-certs\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668908 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-node-log\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668945 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-device-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668944 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29151040-d2de-4a3c-b608-dc74c6aeee0e-serviceca\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-hostroot\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668980 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e137e152-115f-4459-80fe-a92048049da2-konnectivity-ca\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.671597 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.668986 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-var-lib-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-sys-fs\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmsj\" (UniqueName: \"kubernetes.io/projected/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kube-api-access-9dmsj\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669159 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669211 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-config\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bgp\" (UniqueName: \"kubernetes.io/projected/930232e3-20a2-4134-99ab-9817dede3186-kube-api-access-97bgp\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-socket-dir-parent\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lxl\" (UniqueName: \"kubernetes.io/projected/5cf30068-f986-4286-b700-f68bdc0e5f75-kube-api-access-q5lxl\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-ovn\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-netd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669402 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669583 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-registration-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669618 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29151040-d2de-4a3c-b608-dc74c6aeee0e-host\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669652 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cni-binary-copy\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669659 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-sys-fs\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.672217 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.669782 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669853 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-socket-dir-parent\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669894 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-netd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-bin\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.669962 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-hostroot\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.670074 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret podName:7e587446-856d-4580-a69f-94cd5953a81d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:20.170050809 +0000 UTC m=+1.980979705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret") pod "global-pull-secret-syncer-c5d98" (UID: "7e587446-856d-4580-a69f-94cd5953a81d") : object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.670143 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-ovn\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.670458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.670573 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-config\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.670748 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.671360 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovn-node-metrics-cert\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.671415 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-tmp\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.671435 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-tuned\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.672656 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.672124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e137e152-115f-4459-80fe-a92048049da2-agent-certs\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.678955 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.678223 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 01:19:19.678955 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.678246 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 01:19:19.678955 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.678259 2575 projected.go:196] Error preparing data for projected volume kube-api-access-55prj for pod openshift-network-diagnostics/network-check-target-klvct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:19.678955 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:19.678310 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj podName:578616cb-95f4-49ef-afec-65f65cdb2f71 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:20.17829248 +0000 UTC m=+1.989221371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-55prj" (UniqueName: "kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj") pod "network-check-target-klvct" (UID: "578616cb-95f4-49ef-afec-65f65cdb2f71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:19.680699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.680678 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdsp\" (UniqueName: \"kubernetes.io/projected/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-kube-api-access-4zdsp\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.681357 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.681336 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzdq\" (UniqueName: \"kubernetes.io/projected/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-kube-api-access-fdzdq\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.681357 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.681345 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkc6\" (UniqueName: \"kubernetes.io/projected/72564aed-4e32-4815-8a3e-6f61a8c36e83-kube-api-access-lhkc6\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.681357 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.681355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66jj\" (UniqueName: \"kubernetes.io/projected/4af660b2-97ac-4db7-b928-9d660b0c15a9-kube-api-access-z66jj\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.681524 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.681345 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphc2\" (UniqueName: \"kubernetes.io/projected/43b11ef2-886e-4d57-9700-089f51fa360e-kube-api-access-xphc2\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:19.684185 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.684161 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bgp\" (UniqueName: \"kubernetes.io/projected/930232e3-20a2-4134-99ab-9817dede3186-kube-api-access-97bgp\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.688198 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.688172 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmsj\" (UniqueName: \"kubernetes.io/projected/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kube-api-access-9dmsj\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.688291 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.688214 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lxl\" (UniqueName: \"kubernetes.io/projected/5cf30068-f986-4286-b700-f68bdc0e5f75-kube-api-access-q5lxl\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.688472 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.688456 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnjx7\" (UniqueName: \"kubernetes.io/projected/29151040-d2de-4a3c-b608-dc74c6aeee0e-kube-api-access-rnjx7\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.701112 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.701092 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:19:19.712042 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.711899 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:19:19.732091 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.732053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:19.827467 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.827439 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6555262b15c85369c5c728ca31cfdef1.slice/crio-34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344 WatchSource:0}: Error finding container 34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344: Status 404 returned error can't find the container with id 34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344 Oct 31 01:19:19.827855 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.827835 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92683d17f4ef39773bb8bc1b1459025a.slice/crio-cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf WatchSource:0}: Error finding container cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf: Status 404 returned error can't find the container with id cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf Oct 31 01:19:19.832481 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.832467 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:19:19.876109 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.876089 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:19:19.885473 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.885451 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29151040_d2de_4a3c_b608_dc74c6aeee0e.slice/crio-033b32812a0ee616cc654e355a148b6af9da2ab43bbc9361ba963d722fdbf468 WatchSource:0}: Error finding container 033b32812a0ee616cc654e355a148b6af9da2ab43bbc9361ba963d722fdbf468: Status 404 returned error can't find the container with id 033b32812a0ee616cc654e355a148b6af9da2ab43bbc9361ba963d722fdbf468 Oct 31 01:19:19.898798 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.898782 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xfqs2" Oct 31 01:19:19.906605 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.906586 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa8037e_1fcd_4d2f_b8d5_edc99be74be9.slice/crio-641250cc82e7a1a2a56baf6254d325710dc76f7c6ac3ac99f66e3dbc4a961382 WatchSource:0}: Error finding container 641250cc82e7a1a2a56baf6254d325710dc76f7c6ac3ac99f66e3dbc4a961382: Status 404 returned error can't find the container with id 641250cc82e7a1a2a56baf6254d325710dc76f7c6ac3ac99f66e3dbc4a961382 Oct 31 01:19:19.914551 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.914535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:19.922057 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.922039 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode137e152_115f_4459_80fe_a92048049da2.slice/crio-b5bfe3d8199224859cc7ff5a1cd6bfbab661a987ac0c45435dcaf395df1c53a8 WatchSource:0}: Error finding container b5bfe3d8199224859cc7ff5a1cd6bfbab661a987ac0c45435dcaf395df1c53a8: Status 404 returned error can't find the container with id b5bfe3d8199224859cc7ff5a1cd6bfbab661a987ac0c45435dcaf395df1c53a8 Oct 31 01:19:19.932089 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.932075 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:19:19.939638 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.939615 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6ce2e8_64af_45dd_a942_7bbb99bfdb2c.slice/crio-49a684c2a440a507c27e2dbd30a18d4e943ee9f1165d9489437bf7e3a467b6b6 WatchSource:0}: Error finding container 49a684c2a440a507c27e2dbd30a18d4e943ee9f1165d9489437bf7e3a467b6b6: Status 404 returned error can't find the container with id 49a684c2a440a507c27e2dbd30a18d4e943ee9f1165d9489437bf7e3a467b6b6 Oct 31 01:19:19.944025 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.944008 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:19:19.944099 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.944049 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:19.952455 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.952434 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf30068_f986_4286_b700_f68bdc0e5f75.slice/crio-3507825afa99e8cacc91371ac681a6d79094cb59d28321832fabfe90ba58bf26 WatchSource:0}: Error finding container 3507825afa99e8cacc91371ac681a6d79094cb59d28321832fabfe90ba58bf26: Status 404 returned error can't find the container with id 3507825afa99e8cacc91371ac681a6d79094cb59d28321832fabfe90ba58bf26 Oct 31 01:19:19.960389 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:19.960373 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mz75q" Oct 31 01:19:19.969170 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:19.969148 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930232e3_20a2_4134_99ab_9817dede3186.slice/crio-3a6c74706ec1df7d4df231c3bcae8579396870d64aac27547a1bb6b7f8a87986 WatchSource:0}: Error finding container 3a6c74706ec1df7d4df231c3bcae8579396870d64aac27547a1bb6b7f8a87986: Status 404 returned error can't find the container with id 3a6c74706ec1df7d4df231c3bcae8579396870d64aac27547a1bb6b7f8a87986 Oct 31 01:19:20.116428 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:20.116395 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af660b2_97ac_4db7_b928_9d660b0c15a9.slice/crio-8a91720bbb13b8af47c5132c2e04bb1d68c3696a9c1033f726569ad4a408db50 WatchSource:0}: Error finding container 8a91720bbb13b8af47c5132c2e04bb1d68c3696a9c1033f726569ad4a408db50: Status 404 returned error can't find the container with id 8a91720bbb13b8af47c5132c2e04bb1d68c3696a9c1033f726569ad4a408db50 Oct 31 01:19:20.174520 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.174470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:20.174520 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.174503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:20.174677 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.174627 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:20.174677 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.174633 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:20.174769 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.174679 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs podName:43b11ef2-886e-4d57-9700-089f51fa360e nodeName:}" failed. No retries permitted until 2026-10-31 01:19:21.174662912 +0000 UTC m=+2.985591803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs") pod "network-metrics-daemon-wff52" (UID: "43b11ef2-886e-4d57-9700-089f51fa360e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:20.174769 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.174692 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret podName:7e587446-856d-4580-a69f-94cd5953a81d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:21.174686486 +0000 UTC m=+2.985615377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret") pod "global-pull-secret-syncer-c5d98" (UID: "7e587446-856d-4580-a69f-94cd5953a81d") : object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:20.225919 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:20.225896 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72564aed_4e32_4815_8a3e_6f61a8c36e83.slice/crio-9bfad8dacd906348710ca485b77d8c72171891d94d3b397743397c2029eab74e WatchSource:0}: Error finding container 9bfad8dacd906348710ca485b77d8c72171891d94d3b397743397c2029eab74e: Status 404 returned error can't find the container with id 9bfad8dacd906348710ca485b77d8c72171891d94d3b397743397c2029eab74e Oct 31 01:19:20.227127 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:20.227105 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd51cb1_85bf_472e_b0d7_14b6bf1cf591.slice/crio-953f12dadbd965c75879b1c4ebaa5f5d57833f8ce929efdbc36699e769f68afa WatchSource:0}: Error finding container 953f12dadbd965c75879b1c4ebaa5f5d57833f8ce929efdbc36699e769f68afa: Status 404 returned error can't find the container with id 953f12dadbd965c75879b1c4ebaa5f5d57833f8ce929efdbc36699e769f68afa Oct 31 01:19:20.274968 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.274947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:20.275069 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.275058 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 01:19:20.275109 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.275075 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 01:19:20.275109 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.275083 2575 projected.go:196] Error preparing data for projected volume kube-api-access-55prj for pod openshift-network-diagnostics/network-check-target-klvct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:20.275184 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:20.275123 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj podName:578616cb-95f4-49ef-afec-65f65cdb2f71 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:21.275110974 +0000 UTC m=+3.086039865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-55prj" (UniqueName: "kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj") pod "network-check-target-klvct" (UID: "578616cb-95f4-49ef-afec-65f65cdb2f71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:20.589095 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.589061 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:20.646574 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.646515 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pmc4f" event={"ID":"29151040-d2de-4a3c-b608-dc74c6aeee0e","Type":"ContainerStarted","Data":"033b32812a0ee616cc654e355a148b6af9da2ab43bbc9361ba963d722fdbf468"} Oct 31 01:19:20.648340 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.648280 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344"} Oct 31 01:19:20.649575 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.649548 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf"} Oct 31 01:19:20.651114 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.651088 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcqc8" event={"ID":"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591","Type":"ContainerStarted","Data":"953f12dadbd965c75879b1c4ebaa5f5d57833f8ce929efdbc36699e769f68afa"} Oct 31 01:19:20.652534 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.652450 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"9bfad8dacd906348710ca485b77d8c72171891d94d3b397743397c2029eab74e"} Oct 31 01:19:20.654074 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.654051 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerStarted","Data":"8a91720bbb13b8af47c5132c2e04bb1d68c3696a9c1033f726569ad4a408db50"} Oct 31 01:19:20.655290 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.655264 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7rq4g" event={"ID":"e137e152-115f-4459-80fe-a92048049da2","Type":"ContainerStarted","Data":"b5bfe3d8199224859cc7ff5a1cd6bfbab661a987ac0c45435dcaf395df1c53a8"} Oct 31 01:19:20.657183 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.657156 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xfqs2" event={"ID":"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9","Type":"ContainerStarted","Data":"641250cc82e7a1a2a56baf6254d325710dc76f7c6ac3ac99f66e3dbc4a961382"} Oct 31 01:19:20.657520 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.657469 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:19 +0000 UTC" deadline="2028-07-18 22:48:00.860309186 +0000 UTC" Oct 31 01:19:20.657520 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.657521 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15045h28m40.202794171s" Oct 31 01:19:20.658675 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.658618 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mz75q" event={"ID":"930232e3-20a2-4134-99ab-9817dede3186","Type":"ContainerStarted","Data":"3a6c74706ec1df7d4df231c3bcae8579396870d64aac27547a1bb6b7f8a87986"} Oct 31 01:19:20.661115 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.660550 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bf87z" event={"ID":"5cf30068-f986-4286-b700-f68bdc0e5f75","Type":"ContainerStarted","Data":"3507825afa99e8cacc91371ac681a6d79094cb59d28321832fabfe90ba58bf26"} Oct 31 01:19:20.662667 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:20.662644 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" event={"ID":"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c","Type":"ContainerStarted","Data":"49a684c2a440a507c27e2dbd30a18d4e943ee9f1165d9489437bf7e3a467b6b6"} Oct 31 01:19:21.182076 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.181777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:21.182076 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.181838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:21.182076 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.181991 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:21.182076 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.182003 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:21.182076 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.182063 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs podName:43b11ef2-886e-4d57-9700-089f51fa360e nodeName:}" failed. No retries permitted until 2026-10-31 01:19:23.182045882 +0000 UTC m=+4.992974790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs") pod "network-metrics-daemon-wff52" (UID: "43b11ef2-886e-4d57-9700-089f51fa360e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:21.182076 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.182084 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret podName:7e587446-856d-4580-a69f-94cd5953a81d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:23.18207345 +0000 UTC m=+4.993002345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret") pod "global-pull-secret-syncer-c5d98" (UID: "7e587446-856d-4580-a69f-94cd5953a81d") : object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:21.285058 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.285020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:21.285279 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.285203 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 01:19:21.285279 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.285226 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 01:19:21.285279 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.285239 2575 projected.go:196] Error preparing data for projected volume kube-api-access-55prj for pod openshift-network-diagnostics/network-check-target-klvct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:21.285389 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.285301 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj podName:578616cb-95f4-49ef-afec-65f65cdb2f71 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:23.285279785 +0000 UTC m=+5.096208690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-55prj" (UniqueName: "kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj") pod "network-check-target-klvct" (UID: "578616cb-95f4-49ef-afec-65f65cdb2f71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:21.642246 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.641847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:21.642917 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.642404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:21.642917 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.642508 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:21.642917 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.642579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:21.642917 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.642638 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:21.642917 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:21.642664 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:21.658234 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.658179 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:19 +0000 UTC" deadline="2028-06-27 22:09:42.994019066 +0000 UTC" Oct 31 01:19:21.658234 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:21.658234 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14540h50m21.335789243s" Oct 31 01:19:23.199794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:23.199080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:23.199794 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:23.199129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:23.199794 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.199278 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:23.199794 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.199337 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs podName:43b11ef2-886e-4d57-9700-089f51fa360e nodeName:}" failed. No retries permitted until 2026-10-31 01:19:27.199318466 +0000 UTC m=+9.010247370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs") pod "network-metrics-daemon-wff52" (UID: "43b11ef2-886e-4d57-9700-089f51fa360e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:23.199794 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.199697 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:23.199794 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.199762 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret podName:7e587446-856d-4580-a69f-94cd5953a81d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:27.199747656 +0000 UTC m=+9.010676561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret") pod "global-pull-secret-syncer-c5d98" (UID: "7e587446-856d-4580-a69f-94cd5953a81d") : object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:23.301004 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:23.300462 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:23.301004 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.300699 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 01:19:23.301004 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.300741 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 01:19:23.301004 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.300755 2575 projected.go:196] Error preparing data for projected volume kube-api-access-55prj for pod openshift-network-diagnostics/network-check-target-klvct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:23.301004 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.300810 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj podName:578616cb-95f4-49ef-afec-65f65cdb2f71 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:27.300791715 +0000 UTC m=+9.111720626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-55prj" (UniqueName: "kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj") pod "network-check-target-klvct" (UID: "578616cb-95f4-49ef-afec-65f65cdb2f71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:23.641833 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:23.641546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:23.641833 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.641686 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:23.642318 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:23.642073 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:23.642318 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.642164 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:23.642318 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:23.642208 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:23.642318 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:23.642278 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:24.686063 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:24.685180 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerStarted","Data":"3b2fd26ef884fcb409e8493f6b1b9fbd75bab8fb713c8a98e42f91275b38fee0"} Oct 31 01:19:24.709662 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:24.708992 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"4ab65acd0d1debd956d57d811c58177a29b03e43a22a1d924a5141436afadcbd"} Oct 31 01:19:25.642857 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.641297 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:25.642857 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:25.641424 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:25.642857 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.641830 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:25.642857 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:25.641931 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:25.642857 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.642734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:25.642857 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:25.642817 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:25.714692 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.714654 2575 generic.go:356] "Generic (PLEG): container finished" podID="92683d17f4ef39773bb8bc1b1459025a" containerID="4ab65acd0d1debd956d57d811c58177a29b03e43a22a1d924a5141436afadcbd" exitCode=0 Oct 31 01:19:25.715165 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.714783 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerDied","Data":"4ab65acd0d1debd956d57d811c58177a29b03e43a22a1d924a5141436afadcbd"} Oct 31 01:19:25.718235 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.718177 2575 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="3b2fd26ef884fcb409e8493f6b1b9fbd75bab8fb713c8a98e42f91275b38fee0" exitCode=0 Oct 31 01:19:25.718335 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.718250 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerDied","Data":"3b2fd26ef884fcb409e8493f6b1b9fbd75bab8fb713c8a98e42f91275b38fee0"} Oct 31 01:19:25.722022 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.721997 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7rq4g" event={"ID":"e137e152-115f-4459-80fe-a92048049da2","Type":"ContainerStarted","Data":"5f9d0eb09a26bd18320d3a71c1d37868243f047db864fc7e145b6966c609273e"} Oct 31 01:19:25.726165 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.726142 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"a09dffb959e1d2fc35bb443f526373c0b67a163f599523498344b922dd35e20e"} Oct 31 01:19:25.783317 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:25.783264 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" podStartSLOduration=6.783244597 podStartE2EDuration="6.783244597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:25.782283156 +0000 UTC m=+7.593212072" watchObservedRunningTime="2026-10-31 01:19:25.783244597 +0000 UTC m=+7.594173511" Oct 31 01:19:26.732291 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:26.732207 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" event={"ID":"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c","Type":"ContainerStarted","Data":"82780a9cd0cfeb589c4c942fe36650dc6ac00a9a9c8f371ffffe644cf1071828"} Oct 31 01:19:26.735012 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:26.734323 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pmc4f" event={"ID":"29151040-d2de-4a3c-b608-dc74c6aeee0e","Type":"ContainerStarted","Data":"d4aa4d600a14c2a05c308a97f649b78d9bfb75633c397670857cf26cfba69498"} Oct 31 01:19:26.740975 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:26.740575 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"2652a7b235c1e80a3bbd8e448e296f345a76e8b40a75349a743da54ebfb227f1"} Oct 31 01:19:26.756279 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:26.754873 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7rq4g" podStartSLOduration=3.96774609 podStartE2EDuration="8.754854834s" totalImagesPullingTime="4.787108744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:25.802168147 +0000 UTC m=+7.613097062" watchObservedRunningTime="2026-10-31 01:19:26.754854834 +0000 UTC m=+8.565783747" Oct 31 01:19:26.780450 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:26.780400 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" podStartSLOduration=7.780384157 podStartE2EDuration="7.780384157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:26.779581697 +0000 UTC m=+8.590510611" watchObservedRunningTime="2026-10-31 01:19:26.780384157 +0000 UTC m=+8.591313074" Oct 31 01:19:26.780567 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:26.780497 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pmc4f" podStartSLOduration=3.026751804 podStartE2EDuration="8.780486564s" totalImagesPullingTime="5.75373476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:26.756106792 +0000 UTC m=+8.567035707" watchObservedRunningTime="2026-10-31 01:19:26.780486564 +0000 UTC m=+8.591415478" Oct 31 01:19:27.235078 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.235038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:27.235241 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.235095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:27.235241 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.235223 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:27.235359 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.235281 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs podName:43b11ef2-886e-4d57-9700-089f51fa360e nodeName:}" failed. No retries permitted until 2026-10-31 01:19:35.235263722 +0000 UTC m=+17.046192629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs") pod "network-metrics-daemon-wff52" (UID: "43b11ef2-886e-4d57-9700-089f51fa360e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:27.235707 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.235687 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:27.235843 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.235757 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret podName:7e587446-856d-4580-a69f-94cd5953a81d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:35.235742265 +0000 UTC m=+17.046671172 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret") pod "global-pull-secret-syncer-c5d98" (UID: "7e587446-856d-4580-a69f-94cd5953a81d") : object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:27.335681 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.335602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:27.335849 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.335772 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 01:19:27.335849 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.335791 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 01:19:27.335849 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.335802 2575 projected.go:196] Error preparing data for projected volume kube-api-access-55prj for pod openshift-network-diagnostics/network-check-target-klvct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:27.336012 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.335857 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj podName:578616cb-95f4-49ef-afec-65f65cdb2f71 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:35.335838103 +0000 UTC m=+17.146767007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-55prj" (UniqueName: "kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj") pod "network-check-target-klvct" (UID: "578616cb-95f4-49ef-afec-65f65cdb2f71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:27.641671 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.641639 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:27.641849 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.641788 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:27.642213 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.642183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:27.642319 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.642291 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:27.642458 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.642369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:27.642458 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:27.642442 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:27.770139 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.770111 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:27.771430 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:27.771130 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:28.148451 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.148421 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 01:19:28.640764 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.639921 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:19:28.148447575Z","UUID":"c8a64d79-e095-44a1-879b-19b05b4b010c","Handler":null,"Name":"","Endpoint":""} Oct 31 01:19:28.644635 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.644391 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 01:19:28.644635 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.644420 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 01:19:28.745479 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.745446 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xfqs2" event={"ID":"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9","Type":"ContainerStarted","Data":"01d726c992fd2219aaa7d56956c3b0505a30207c6dc0528624e6526b9581a219"} Oct 31 01:19:28.749181 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.748875 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" event={"ID":"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c","Type":"ContainerStarted","Data":"14e42194b7b68eaeef4f231f40903769dc67cc4c9f71b671a1fb886ff40a97dc"} Oct 31 01:19:28.749181 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.748937 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:28.749456 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.749437 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:19:28.769605 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:28.769568 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xfqs2" podStartSLOduration=2.699947858 podStartE2EDuration="10.769556756s" totalImagesPullingTime="8.069608898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:28.769084053 +0000 UTC m=+10.580012965" watchObservedRunningTime="2026-10-31 01:19:28.769556756 +0000 UTC m=+10.580485669" Oct 31 01:19:29.641608 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.641548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:29.641922 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:29.641682 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:29.642092 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.642072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:29.642204 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:29.642184 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:29.642269 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.642255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:29.642350 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:29.642330 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:29.753199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.753166 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerStarted","Data":"6e3dfee6702f03a54075282d331802fc604a09cd6e70ce4a1cc3d76501343b37"} Oct 31 01:19:29.754943 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.754834 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mz75q" event={"ID":"930232e3-20a2-4134-99ab-9817dede3186","Type":"ContainerStarted","Data":"498c50b383ac440e3c20598228cb03c33ad7dc984fc01ea1b9458e7eacd40328"} Oct 31 01:19:29.758413 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.758387 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" event={"ID":"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c","Type":"ContainerStarted","Data":"c91e65c401899622cd137b2fc7351d7a4eff8ed7615bcceb09f56ae6e34e6af0"} Oct 31 01:19:29.807557 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.807511 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" podStartSLOduration=2.721374367 podStartE2EDuration="11.807498539s" totalImagesPullingTime="9.086124172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:29.807159987 +0000 UTC m=+11.618088900" watchObservedRunningTime="2026-10-31 01:19:29.807498539 +0000 UTC m=+11.618427452" Oct 31 01:19:29.825745 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:29.825695 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mz75q" podStartSLOduration=3.214792043 podStartE2EDuration="11.825685268s" totalImagesPullingTime="8.610893225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:29.825055838 +0000 UTC m=+11.635984934" watchObservedRunningTime="2026-10-31 01:19:29.825685268 +0000 UTC m=+11.636614181" Oct 31 01:19:30.761169 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:30.760914 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcqc8" event={"ID":"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591","Type":"ContainerStarted","Data":"391e90d83599c5ed544e0529917f662b0265ee944533c9c8297e789686721bfc"} Oct 31 01:19:30.763033 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:30.763009 2575 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="6e3dfee6702f03a54075282d331802fc604a09cd6e70ce4a1cc3d76501343b37" exitCode=0 Oct 31 01:19:30.763188 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:30.763165 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerDied","Data":"6e3dfee6702f03a54075282d331802fc604a09cd6e70ce4a1cc3d76501343b37"} Oct 31 01:19:30.779950 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:30.779908 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fcqc8" podStartSLOduration=3.311688212 podStartE2EDuration="11.779894958s" totalImagesPullingTime="8.468206746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:30.779790198 +0000 UTC m=+12.590719112" watchObservedRunningTime="2026-10-31 01:19:30.779894958 +0000 UTC m=+12.590823873" Oct 31 01:19:31.641991 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:31.641950 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:31.641991 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:31.641973 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:31.641991 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:31.641993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:31.642228 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:31.642088 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:31.642228 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:31.642188 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:31.642323 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:31.642270 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:31.766604 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:31.766571 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bf87z" event={"ID":"5cf30068-f986-4286-b700-f68bdc0e5f75","Type":"ContainerStarted","Data":"e9d26f5167d4ea4caede47a82c95e196724d0d521942a8ab5566eb9d56dec46e"} Oct 31 01:19:32.771183 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:32.771143 2575 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="e8281205d302897656d8ce6576ecfa64cacfaa09396306c91d139262e9ac6dcd" exitCode=0 Oct 31 01:19:32.771860 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:32.771252 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerDied","Data":"e8281205d302897656d8ce6576ecfa64cacfaa09396306c91d139262e9ac6dcd"} Oct 31 01:19:32.802937 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:32.802890 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bf87z" podStartSLOduration=3.918118778 podStartE2EDuration="14.802870158s" totalImagesPullingTime="10.88475138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:31.791505283 +0000 UTC m=+13.602434198" watchObservedRunningTime="2026-10-31 01:19:32.802870158 +0000 UTC m=+14.613799070" Oct 31 01:19:33.641114 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:33.641088 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:33.641298 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:33.641193 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:33.641298 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:33.641206 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:33.641298 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:33.641234 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:33.641460 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:33.641315 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:33.641460 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:33.641397 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:34.779377 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:34.779201 2575 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="3f50a2fbcb84affe7d79e03b51001caa779d29d4f42edc1dc4c1b8e692fa11f1" exitCode=0 Oct 31 01:19:34.779377 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:34.779290 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerDied","Data":"3f50a2fbcb84affe7d79e03b51001caa779d29d4f42edc1dc4c1b8e692fa11f1"} Oct 31 01:19:35.298880 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.298856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:35.298985 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.298903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:35.298985 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.298967 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:35.299087 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.298995 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:35.299087 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.299035 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret podName:7e587446-856d-4580-a69f-94cd5953a81d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.299016998 +0000 UTC m=+33.109945913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret") pod "global-pull-secret-syncer-c5d98" (UID: "7e587446-856d-4580-a69f-94cd5953a81d") : object "kube-system"/"original-pull-secret" not registered Oct 31 01:19:35.299087 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.299054 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs podName:43b11ef2-886e-4d57-9700-089f51fa360e nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.299044816 +0000 UTC m=+33.109973707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs") pod "network-metrics-daemon-wff52" (UID: "43b11ef2-886e-4d57-9700-089f51fa360e") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 01:19:35.399868 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.399848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:35.400013 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.399997 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 01:19:35.400053 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.400019 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 01:19:35.400053 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.400029 2575 projected.go:196] Error preparing data for projected volume kube-api-access-55prj for pod openshift-network-diagnostics/network-check-target-klvct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:35.400123 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.400072 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj podName:578616cb-95f4-49ef-afec-65f65cdb2f71 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.400056732 +0000 UTC m=+33.210985636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-55prj" (UniqueName: "kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj") pod "network-check-target-klvct" (UID: "578616cb-95f4-49ef-afec-65f65cdb2f71") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 01:19:35.641613 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.641400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:35.641790 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.641667 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:35.641955 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.641930 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:35.642073 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.642049 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:35.642073 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.642065 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:35.642158 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:35.642149 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:35.785293 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785258 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:19:35.785730 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785651 2575 generic.go:356] "Generic (PLEG): container finished" podID="72564aed-4e32-4815-8a3e-6f61a8c36e83" containerID="2c9576f4d1933fdfa68e83b30f3f03d125afc002f39d3ce0abc79cbbdb5bbc84" exitCode=1 Oct 31 01:19:35.785730 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785700 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"fbd670e4760dece2915e3b82dfe01f067dc81ec027bc6f78daba7863d84d422c"} Oct 31 01:19:35.785818 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785791 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"1af80a00fb0bf959aacf5bf18dd5f0c579a68a7e81097a78ad5e29273b7e4bb3"} Oct 31 01:19:35.785855 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785840 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"5975239eb3d095e6660fb364fdf726f3585895d34baa31a156856695789bd385"} Oct 31 01:19:35.785902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785887 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"5f4705919d79c328e2ab30600605568dea6ca6d83f1a3ac742695e7b0f4e8ca1"} Oct 31 01:19:35.785937 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785931 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerDied","Data":"2c9576f4d1933fdfa68e83b30f3f03d125afc002f39d3ce0abc79cbbdb5bbc84"} Oct 31 01:19:35.785997 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:35.785984 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"b94bb71fd6664bea7e3aeb1ee2a70a1e4913c8e1d69efc642dddc25e02aa2f44"} Oct 31 01:19:36.790129 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:36.790103 2575 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="09a9e2f642d305e0fd11f1c7027e2578fe201c2c57c6d7eab84d0cfed1b3d4f3" exitCode=0 Oct 31 01:19:36.790544 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:36.790140 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerDied","Data":"09a9e2f642d305e0fd11f1c7027e2578fe201c2c57c6d7eab84d0cfed1b3d4f3"} Oct 31 01:19:37.641785 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:37.641761 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:37.641785 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:37.641781 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:37.641921 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:37.641873 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:37.641921 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:37.641906 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:37.641998 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:37.641956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:37.642055 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:37.642041 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:37.795289 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:37.795268 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:19:37.795662 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:37.795639 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"dcc05d37e96fc5d5c62addc854e736b248d743017b328547eefa116137158062"} Oct 31 01:19:37.798279 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:37.798252 2575 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="8c93f3d9d0b3c7bc7602b70aab4e95762fee5498cf8e6e805cd976a1f007e82f" exitCode=0 Oct 31 01:19:37.798367 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:37.798288 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerDied","Data":"8c93f3d9d0b3c7bc7602b70aab4e95762fee5498cf8e6e805cd976a1f007e82f"} Oct 31 01:19:38.803842 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:38.803810 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h7k55" event={"ID":"4af660b2-97ac-4db7-b928-9d660b0c15a9","Type":"ContainerStarted","Data":"f82f18242abeb52e56acf343b1cde53dc2a32071f994fecf1949a7bedfd85357"} Oct 31 01:19:38.835463 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:38.835418 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h7k55" podStartSLOduration=8.156241975 podStartE2EDuration="19.835407175s" totalImagesPullingTime="11.6791652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:38.834956993 +0000 UTC m=+20.645885905" watchObservedRunningTime="2026-10-31 01:19:38.835407175 +0000 UTC m=+20.646336089" Oct 31 01:19:39.641327 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:39.641296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:39.641327 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:39.641317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:39.641533 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:39.641374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:39.641533 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:39.641452 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:39.641533 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:39.641506 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:39.641740 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:39.641599 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:39.810600 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:39.810574 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:19:39.811164 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:39.811000 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"ce21646e373e59411778634190f7fd595c37be0ace7195b30e5b8d2f4c252ce3"} Oct 31 01:19:39.811697 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:39.811676 2575 scope.go:122] "RemoveContainer" containerID="2c9576f4d1933fdfa68e83b30f3f03d125afc002f39d3ce0abc79cbbdb5bbc84" Oct 31 01:19:40.816436 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.816269 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:19:40.816951 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.816777 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" event={"ID":"72564aed-4e32-4815-8a3e-6f61a8c36e83","Type":"ContainerStarted","Data":"a1e6f7b37caec3cae154b650f7de545e2d1db77ab8006a5810b06b22cb8eda91"} Oct 31 01:19:40.817010 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.816987 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:40.817076 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.817064 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:40.817134 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.817121 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:40.832522 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.832502 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:40.832940 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.832926 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:19:40.862181 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:40.862128 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" podStartSLOduration=7.022849592 podStartE2EDuration="21.862111062s" totalImagesPullingTime="14.83926147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:40.861770539 +0000 UTC m=+22.672699453" watchObservedRunningTime="2026-10-31 01:19:40.862111062 +0000 UTC m=+22.673039976" Oct 31 01:19:41.624836 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:41.624805 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c5d98"] Oct 31 01:19:41.625056 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:41.624987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:41.625108 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:41.625082 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:41.627604 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:41.627584 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-klvct"] Oct 31 01:19:41.627689 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:41.627663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:41.627781 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:41.627761 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:41.630922 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:41.630899 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wff52"] Oct 31 01:19:41.631034 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:41.631012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:41.631151 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:41.631115 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:43.641711 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:43.641675 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:43.642146 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:43.641674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:43.642146 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:43.641807 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:43.642146 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:43.641677 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:43.642146 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:43.641884 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:43.642146 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:43.641913 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:45.641527 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:45.641496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:45.642071 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:45.641583 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-klvct" podUID="578616cb-95f4-49ef-afec-65f65cdb2f71" Oct 31 01:19:45.642071 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:45.641631 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:45.642071 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:45.641736 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wff52" podUID="43b11ef2-886e-4d57-9700-089f51fa360e" Oct 31 01:19:45.642071 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:45.641784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:45.642071 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:45.641883 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c5d98" podUID="7e587446-856d-4580-a69f-94cd5953a81d" Oct 31 01:19:47.527520 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.527497 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeReady" Oct 31 01:19:47.527897 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.527624 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 01:19:47.567584 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.567558 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d9zxv"] Oct 31 01:19:47.572853 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.572827 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bcgq4"] Oct 31 01:19:47.572996 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.572926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" Oct 31 01:19:47.575690 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.575670 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fb9bcc569-v7h6k"] Oct 31 01:19:47.575823 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.575797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" Oct 31 01:19:47.581869 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.581848 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54d575cd44-5jzvn"] Oct 31 01:19:47.582042 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.582016 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.584658 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.584623 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.585105 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.585088 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg"] Oct 31 01:19:47.585207 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.585191 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.585432 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.585326 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.590400 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.590385 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7hl7l"] Oct 31 01:19:47.590530 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.590516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.593934 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.593914 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54786d95-fr8g9"] Oct 31 01:19:47.594052 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.594038 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.594113 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.594062 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 01:19:47.594183 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.594164 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wq6mh\"" Oct 31 01:19:47.594273 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.594253 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 01:19:47.595865 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.595848 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 01:19:47.596357 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.596343 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr"] Oct 31 01:19:47.596493 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.596447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.600672 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.600655 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7"] Oct 31 01:19:47.600847 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.600780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.604524 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.604503 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zkw9h"] Oct 31 01:19:47.605267 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.604706 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" Oct 31 01:19:47.610025 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.609860 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-llbl2"] Oct 31 01:19:47.610564 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.610480 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.614011 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.613994 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf"] Oct 31 01:19:47.614156 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.614131 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-llbl2" Oct 31 01:19:47.617643 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.617622 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452"] Oct 31 01:19:47.617766 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.617749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.622114 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.622067 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-d74bn"] Oct 31 01:19:47.624188 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.623844 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.629473 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.628927 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj"] Oct 31 01:19:47.629473 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.629402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:47.632744 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.632182 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d9zxv"] Oct 31 01:19:47.632744 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.632223 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bcgq4"] Oct 31 01:19:47.632744 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.632259 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mrqzl"] Oct 31 01:19:47.632744 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.632598 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" Oct 31 01:19:47.638980 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.638636 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jn8kf"] Oct 31 01:19:47.639187 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.639172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.643211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643195 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7"] Oct 31 01:19:47.643303 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643224 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54d575cd44-5jzvn"] Oct 31 01:19:47.643303 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643288 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg"] Oct 31 01:19:47.643303 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643296 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643317 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-llbl2"] Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643332 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb9bcc569-v7h6k"] Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643338 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643345 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452"] Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643358 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj"] Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643377 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54786d95-fr8g9"] Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643393 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7hl7l"] Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643410 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jn8kf"] Oct 31 01:19:47.643414 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643422 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrqzl"] Oct 31 01:19:47.643931 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643444 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf"] Oct 31 01:19:47.643931 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643459 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr"] Oct 31 01:19:47.643931 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643475 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zkw9h"] Oct 31 01:19:47.643931 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.643493 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-d74bn"] Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683497 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683504 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683498 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683504 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683564 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683572 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683580 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683600 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683604 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683616 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683606 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683639 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683645 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683652 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683575 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683660 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683655 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683682 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683697 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683740 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.684211 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.683762 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.685755 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.685742 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.685910 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.685892 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.685995 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.685917 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.685995 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.685924 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:47.685995 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.685937 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 01:19:47.685995 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.685947 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.686131 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.686013 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 01:19:47.691009 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.690992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.691086 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.691536 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxmkv\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.691574 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.691574 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.691629 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:47.691629 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-tmp\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.691699 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.691747 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.691799 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-service-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.691799 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.691904 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691810 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.691904 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691843 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a3cb44a-89d0-4bb6-9815-3f52494b3047-config\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.691904 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691865 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f35a7382-ce08-44b1-8427-33a5d30252b7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.691904 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-snapshots\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.692093 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691931 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.692093 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b85121a-782e-4f5a-b984-a775a806b231-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.692093 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.691997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8280d35c-b526-41e0-b4a2-0bd74b181789-serving-cert\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.692093 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-config\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.692093 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb74w\" (UniqueName: \"kubernetes.io/projected/3e5f8170-98cf-4487-84e2-0aeccb45ba6c-kube-api-access-bb74w\") pod \"volume-data-source-validator-58c9f5f46d-q2bv7\" (UID: \"3e5f8170-98cf-4487-84e2-0aeccb45ba6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f35a7382-ce08-44b1-8427-33a5d30252b7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgdm\" (UniqueName: \"kubernetes.io/projected/f35a7382-ce08-44b1-8427-33a5d30252b7-kube-api-access-rzgdm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-default-certificate\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692227 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pxs\" (UniqueName: \"kubernetes.io/projected/9fb8224e-3136-4261-8344-c4bc50dee44c-kube-api-access-s5pxs\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692291 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgq5\" (UniqueName: \"kubernetes.io/projected/f1b7733e-21c3-4770-bb58-d538802cbcaf-kube-api-access-9bgq5\") pod \"kas-connection-checker-685895c7c9-d9zxv\" (UID: \"f1b7733e-21c3-4770-bb58-d538802cbcaf\") " pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" Oct 31 01:19:47.692331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692309 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssr7p\" (UniqueName: \"kubernetes.io/projected/5e6f6573-29b9-4b30-a82c-2b6e8948583f-kube-api-access-ssr7p\") pod \"network-check-source-6f78d59f99-dgzjj\" (UID: \"5e6f6573-29b9-4b30-a82c-2b6e8948583f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2jhb\" (UniqueName: \"kubernetes.io/projected/22fd2129-24b4-4c50-91be-bea78425e824-kube-api-access-d2jhb\") pod \"kas-connection-checker-685895c7c9-llbl2\" (UID: \"22fd2129-24b4-4c50-91be-bea78425e824\") " pod="kube-system/kas-connection-checker-685895c7c9-llbl2" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmbh\" (UniqueName: \"kubernetes.io/projected/6b85121a-782e-4f5a-b984-a775a806b231-kube-api-access-jcmbh\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3cb44a-89d0-4bb6-9815-3f52494b3047-serving-cert\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692531 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zclbj\" (UniqueName: \"kubernetes.io/projected/57ac8842-181e-4186-b7b4-5175a511a0e6-kube-api-access-zclbj\") pod \"kas-connection-checker-685895c7c9-bcgq4\" (UID: \"57ac8842-181e-4186-b7b4-5175a511a0e6\") " pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692705 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-stats-auth\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692769 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkt44\" (UniqueName: \"kubernetes.io/projected/dd04b736-45e6-44cc-a032-e47ef93641a9-kube-api-access-hkt44\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.692784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692787 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9h6\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.693490 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692805 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd04b736-45e6-44cc-a032-e47ef93641a9-serving-cert\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.693490 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692827 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb6xr\" (UniqueName: \"kubernetes.io/projected/8280d35c-b526-41e0-b4a2-0bd74b181789-kube-api-access-nb6xr\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.693490 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.693490 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-trusted-ca\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.693490 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd88h\" (UniqueName: \"kubernetes.io/projected/8a3cb44a-89d0-4bb6-9815-3f52494b3047-kube-api-access-gd88h\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.693490 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.692903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8hh\" (UniqueName: \"kubernetes.io/projected/befed7f0-b23a-48a5-803e-6f8bf380012c-kube-api-access-fp8hh\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.695905 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.695889 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 01:19:47.700835 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.700805 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 01:19:47.700835 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.700823 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 01:19:47.700992 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.700871 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 01:19:47.700992 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.700887 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 01:19:47.702649 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.702632 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 01:19:47.702790 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.702728 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 01:19:47.702865 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.702850 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 01:19:47.702946 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.702935 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 01:19:47.703046 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.703023 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 01:19:47.703214 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.703198 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 01:19:47.703257 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.703215 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 01:19:47.703257 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.703202 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 01:19:47.703257 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.703202 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 01:19:47.705318 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705302 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qvjmz\"" Oct 31 01:19:47.705583 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705568 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xp7zg\"" Oct 31 01:19:47.705583 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705575 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9clxs\"" Oct 31 01:19:47.705703 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705677 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qjxmz\"" Oct 31 01:19:47.705703 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705697 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\"" Oct 31 01:19:47.705840 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705701 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\"" Oct 31 01:19:47.705840 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705753 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\"" Oct 31 01:19:47.705840 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705762 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-whdn9\"" Oct 31 01:19:47.705840 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705815 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cv59q\"" Oct 31 01:19:47.705840 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705817 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ctd5c\"" Oct 31 01:19:47.705840 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705818 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p75wq\"" Oct 31 01:19:47.706025 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705931 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-79nk6\"" Oct 31 01:19:47.706025 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.705991 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q7mlm\"" Oct 31 01:19:47.706082 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.706032 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lqw4r\"" Oct 31 01:19:47.706111 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.706087 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gr6nh\"" Oct 31 01:19:47.708081 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.708067 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 01:19:47.708815 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.708691 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 01:19:47.708815 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.708747 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 01:19:47.708897 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.708875 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 01:19:47.793250 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.793250 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.793250 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793244 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f35a7382-ce08-44b1-8427-33a5d30252b7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.793448 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgdm\" (UniqueName: \"kubernetes.io/projected/f35a7382-ce08-44b1-8427-33a5d30252b7-kube-api-access-rzgdm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.793448 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-default-certificate\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.793448 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pxs\" (UniqueName: \"kubernetes.io/projected/9fb8224e-3136-4261-8344-c4bc50dee44c-kube-api-access-s5pxs\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.793448 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.793448 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgq5\" (UniqueName: \"kubernetes.io/projected/f1b7733e-21c3-4770-bb58-d538802cbcaf-kube-api-access-9bgq5\") pod \"kas-connection-checker-685895c7c9-d9zxv\" (UID: \"f1b7733e-21c3-4770-bb58-d538802cbcaf\") " pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" Oct 31 01:19:47.793448 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.793448 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.793402 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:19:47.793925 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.793488 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.293466648 +0000 UTC m=+30.104395539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:19:47.793925 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.793925 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.793925 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssr7p\" (UniqueName: \"kubernetes.io/projected/5e6f6573-29b9-4b30-a82c-2b6e8948583f-kube-api-access-ssr7p\") pod \"network-check-source-6f78d59f99-dgzjj\" (UID: \"5e6f6573-29b9-4b30-a82c-2b6e8948583f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" Oct 31 01:19:47.793925 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jhb\" (UniqueName: \"kubernetes.io/projected/22fd2129-24b4-4c50-91be-bea78425e824-kube-api-access-d2jhb\") pod \"kas-connection-checker-685895c7c9-llbl2\" (UID: \"22fd2129-24b4-4c50-91be-bea78425e824\") " pod="kube-system/kas-connection-checker-685895c7c9-llbl2" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793951 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.793955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmbh\" (UniqueName: \"kubernetes.io/projected/6b85121a-782e-4f5a-b984-a775a806b231-kube-api-access-jcmbh\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shdj\" (UniqueName: \"kubernetes.io/projected/df649d60-b1cd-408d-a0fc-e74fd86e6896-kube-api-access-8shdj\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3cb44a-89d0-4bb6-9815-3f52494b3047-serving-cert\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794099 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794155 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.794199 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclbj\" (UniqueName: \"kubernetes.io/projected/57ac8842-181e-4186-b7b4-5175a511a0e6-kube-api-access-zclbj\") pod \"kas-connection-checker-685895c7c9-bcgq4\" (UID: \"57ac8842-181e-4186-b7b4-5175a511a0e6\") " pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-stats-auth\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkt44\" (UniqueName: \"kubernetes.io/projected/dd04b736-45e6-44cc-a032-e47ef93641a9-kube-api-access-hkt44\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a489a94c-56c7-45df-92cb-42177dadd9c1-cert\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9h6\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd04b736-45e6-44cc-a032-e47ef93641a9-serving-cert\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794359 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6xr\" (UniqueName: \"kubernetes.io/projected/8280d35c-b526-41e0-b4a2-0bd74b181789-kube-api-access-nb6xr\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794387 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-trusted-ca\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794502 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd88h\" (UniqueName: \"kubernetes.io/projected/8a3cb44a-89d0-4bb6-9815-3f52494b3047-kube-api-access-gd88h\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794531 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8hh\" (UniqueName: \"kubernetes.io/projected/befed7f0-b23a-48a5-803e-6f8bf380012c-kube-api-access-fp8hh\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.794541 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.794618 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.794584 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.294571098 +0000 UTC m=+30.105499989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmkv\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-tmp\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df649d60-b1cd-408d-a0fc-e74fd86e6896-tmp-dir\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794836 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-service-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a3cb44a-89d0-4bb6-9815-3f52494b3047-config\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.794992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f35a7382-ce08-44b1-8427-33a5d30252b7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjkg8\" (UniqueName: \"kubernetes.io/projected/a489a94c-56c7-45df-92cb-42177dadd9c1-kube-api-access-vjkg8\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:47.795408 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-snapshots\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795109 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b85121a-782e-4f5a-b984-a775a806b231-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8280d35c-b526-41e0-b4a2-0bd74b181789-serving-cert\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df649d60-b1cd-408d-a0fc-e74fd86e6896-config-volume\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-config\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb74w\" (UniqueName: \"kubernetes.io/projected/3e5f8170-98cf-4487-84e2-0aeccb45ba6c-kube-api-access-bb74w\") pod \"volume-data-source-validator-58c9f5f46d-q2bv7\" (UID: \"3e5f8170-98cf-4487-84e2-0aeccb45ba6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795351 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.795598 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-trusted-ca\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.796197 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.796030 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.796659 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.796312 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:47.796659 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.796490 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.796659 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.795019 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:47.796659 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.796559 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb9bcc569-v7h6k: secret "image-registry-tls" not found Oct 31 01:19:47.797054 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.796877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-tmp\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.798016 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.797054 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f35a7382-ce08-44b1-8427-33a5d30252b7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.798016 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.798009 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-config\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.797371 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-snapshots\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.797478 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-service-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.797287 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.795070 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.795683 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.295669494 +0000 UTC m=+30.106598408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.795761 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.798141 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54d575cd44-5jzvn: secret "image-registry-tls" not found Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.798208 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls podName:bd51c16f-f327-4939-ad47-4acc6aaf62c2 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.29819388 +0000 UTC m=+30.109122774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls") pod "image-registry-fb9bcc569-v7h6k" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2") : secret "image-registry-tls" not found Oct 31 01:19:47.798334 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.798294 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.298230481 +0000 UTC m=+30.109159371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:47.798842 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.798418 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls podName:b123e4c4-57cd-49f8-985b-158b4deab82d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.298358981 +0000 UTC m=+30.109287872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls") pod "image-registry-54d575cd44-5jzvn" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d") : secret "image-registry-tls" not found Oct 31 01:19:47.798842 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.798433 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.298425463 +0000 UTC m=+30.109354365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:19:47.798842 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.798285 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.798842 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.798460 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-default-certificate\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.799052 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.798898 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-stats-auth\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.799123 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.799097 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.799344 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.799320 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f35a7382-ce08-44b1-8427-33a5d30252b7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.799614 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.799585 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b85121a-782e-4f5a-b984-a775a806b231-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.799744 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.799701 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a3cb44a-89d0-4bb6-9815-3f52494b3047-config\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.799853 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.799760 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3cb44a-89d0-4bb6-9815-3f52494b3047-serving-cert\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.800372 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.800347 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.800775 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.800756 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd04b736-45e6-44cc-a032-e47ef93641a9-serving-cert\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.801010 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.800994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.801149 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.801134 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.801943 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.801927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.802205 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.802190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8280d35c-b526-41e0-b4a2-0bd74b181789-serving-cert\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.809781 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.808224 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pxs\" (UniqueName: \"kubernetes.io/projected/9fb8224e-3136-4261-8344-c4bc50dee44c-kube-api-access-s5pxs\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:47.809781 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.808491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssr7p\" (UniqueName: \"kubernetes.io/projected/5e6f6573-29b9-4b30-a82c-2b6e8948583f-kube-api-access-ssr7p\") pod \"network-check-source-6f78d59f99-dgzjj\" (UID: \"5e6f6573-29b9-4b30-a82c-2b6e8948583f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" Oct 31 01:19:47.809781 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.808938 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgdm\" (UniqueName: \"kubernetes.io/projected/f35a7382-ce08-44b1-8427-33a5d30252b7-kube-api-access-rzgdm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.809781 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.809349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.809781 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.809558 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgq5\" (UniqueName: \"kubernetes.io/projected/f1b7733e-21c3-4770-bb58-d538802cbcaf-kube-api-access-9bgq5\") pod \"kas-connection-checker-685895c7c9-d9zxv\" (UID: \"f1b7733e-21c3-4770-bb58-d538802cbcaf\") " pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" Oct 31 01:19:47.810098 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.810044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8hh\" (UniqueName: \"kubernetes.io/projected/befed7f0-b23a-48a5-803e-6f8bf380012c-kube-api-access-fp8hh\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:47.810479 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.810248 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6xr\" (UniqueName: \"kubernetes.io/projected/8280d35c-b526-41e0-b4a2-0bd74b181789-kube-api-access-nb6xr\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.810479 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.810461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmbh\" (UniqueName: \"kubernetes.io/projected/6b85121a-782e-4f5a-b984-a775a806b231-kube-api-access-jcmbh\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:47.811784 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.811738 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jhb\" (UniqueName: \"kubernetes.io/projected/22fd2129-24b4-4c50-91be-bea78425e824-kube-api-access-d2jhb\") pod \"kas-connection-checker-685895c7c9-llbl2\" (UID: \"22fd2129-24b4-4c50-91be-bea78425e824\") " pod="kube-system/kas-connection-checker-685895c7c9-llbl2" Oct 31 01:19:47.812246 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.812201 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclbj\" (UniqueName: \"kubernetes.io/projected/57ac8842-181e-4186-b7b4-5175a511a0e6-kube-api-access-zclbj\") pod \"kas-connection-checker-685895c7c9-bcgq4\" (UID: \"57ac8842-181e-4186-b7b4-5175a511a0e6\") " pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" Oct 31 01:19:47.812871 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.812846 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd88h\" (UniqueName: \"kubernetes.io/projected/8a3cb44a-89d0-4bb6-9815-3f52494b3047-kube-api-access-gd88h\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.813167 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.813138 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb74w\" (UniqueName: \"kubernetes.io/projected/3e5f8170-98cf-4487-84e2-0aeccb45ba6c-kube-api-access-bb74w\") pod \"volume-data-source-validator-58c9f5f46d-q2bv7\" (UID: \"3e5f8170-98cf-4487-84e2-0aeccb45ba6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" Oct 31 01:19:47.813625 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.813595 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.813958 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.813939 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9h6\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:47.814090 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.814073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkt44\" (UniqueName: \"kubernetes.io/projected/dd04b736-45e6-44cc-a032-e47ef93641a9-kube-api-access-hkt44\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.814857 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.814837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmkv\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:47.891849 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.891823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" Oct 31 01:19:47.896173 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a489a94c-56c7-45df-92cb-42177dadd9c1-cert\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:47.896279 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df649d60-b1cd-408d-a0fc-e74fd86e6896-tmp-dir\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.896331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjkg8\" (UniqueName: \"kubernetes.io/projected/a489a94c-56c7-45df-92cb-42177dadd9c1-kube-api-access-vjkg8\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:47.896392 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df649d60-b1cd-408d-a0fc-e74fd86e6896-config-volume\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.896392 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896376 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.896493 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shdj\" (UniqueName: \"kubernetes.io/projected/df649d60-b1cd-408d-a0fc-e74fd86e6896-kube-api-access-8shdj\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.896543 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.896512 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 01:19:47.896588 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:47.896573 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls podName:df649d60-b1cd-408d-a0fc-e74fd86e6896 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:48.396553914 +0000 UTC m=+30.207482805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls") pod "dns-default-mrqzl" (UID: "df649d60-b1cd-408d-a0fc-e74fd86e6896") : secret "dns-default-metrics-tls" not found Oct 31 01:19:47.896648 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896592 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df649d60-b1cd-408d-a0fc-e74fd86e6896-tmp-dir\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.896939 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.896900 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df649d60-b1cd-408d-a0fc-e74fd86e6896-config-volume\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.898841 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.898819 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a489a94c-56c7-45df-92cb-42177dadd9c1-cert\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:47.899892 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.899876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" Oct 31 01:19:47.911074 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.911048 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shdj\" (UniqueName: \"kubernetes.io/projected/df649d60-b1cd-408d-a0fc-e74fd86e6896-kube-api-access-8shdj\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:47.911250 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.911232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjkg8\" (UniqueName: \"kubernetes.io/projected/a489a94c-56c7-45df-92cb-42177dadd9c1-kube-api-access-vjkg8\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:47.940649 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.940613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:19:47.946680 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.946655 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:19:47.966491 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.966168 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:19:47.968756 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.968567 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" Oct 31 01:19:47.977261 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.976890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:47.981964 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:47.980081 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-llbl2" Oct 31 01:19:48.006073 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.004727 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" Oct 31 01:19:48.034036 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.032733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:19:48.191602 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.191582 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-d9zxv"] Oct 31 01:19:48.194627 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.194598 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b7733e_21c3_4770_bb58_d538802cbcaf.slice/crio-832394168dbe00bc7be0ea1b777b1528a2a81d660ac8fc4b55c020a9da8bed52 WatchSource:0}: Error finding container 832394168dbe00bc7be0ea1b777b1528a2a81d660ac8fc4b55c020a9da8bed52: Status 404 returned error can't find the container with id 832394168dbe00bc7be0ea1b777b1528a2a81d660ac8fc4b55c020a9da8bed52 Oct 31 01:19:48.195477 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.195458 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-bcgq4"] Oct 31 01:19:48.197154 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.197131 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ac8842_181e_4186_b7b4_5175a511a0e6.slice/crio-a36700057e998a5b42a124389eb2d08769bf66d4a0943caa30a279671e469959 WatchSource:0}: Error finding container a36700057e998a5b42a124389eb2d08769bf66d4a0943caa30a279671e469959: Status 404 returned error can't find the container with id a36700057e998a5b42a124389eb2d08769bf66d4a0943caa30a279671e469959 Oct 31 01:19:48.208871 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.208846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:48.298743 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.298111 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr"] Oct 31 01:19:48.302436 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.302407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:48.302539 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.302474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:48.302655 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.302637 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:48.303095 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.303073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:48.303220 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.303171 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.303138909 +0000 UTC m=+31.114067807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:48.303220 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.303190 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.30318037 +0000 UTC m=+31.114109268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:19:48.303350 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.303263 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:48.303350 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.303275 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54d575cd44-5jzvn: secret "image-registry-tls" not found Oct 31 01:19:48.303593 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.303574 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls podName:b123e4c4-57cd-49f8-985b-158b4deab82d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.303528382 +0000 UTC m=+31.114457288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls") pod "image-registry-54d575cd44-5jzvn" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d") : secret "image-registry-tls" not found Oct 31 01:19:48.303663 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.303643 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:48.303807 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.303738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:48.304475 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.304449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:48.304556 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.304490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:48.304657 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304414 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:19:48.304755 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304706 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.304692985 +0000 UTC m=+31.115621894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:19:48.304835 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304799 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:48.304835 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304810 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb9bcc569-v7h6k: secret "image-registry-tls" not found Oct 31 01:19:48.304928 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304841 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls podName:bd51c16f-f327-4939-ad47-4acc6aaf62c2 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.304831227 +0000 UTC m=+31.115760134 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls") pod "image-registry-fb9bcc569-v7h6k" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2") : secret "image-registry-tls" not found Oct 31 01:19:48.304928 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304643 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:19:48.304928 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304872 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.304864287 +0000 UTC m=+31.115793224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:19:48.305037 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304946 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:19:48.305037 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.304974 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.304965067 +0000 UTC m=+31.115893972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:19:48.324579 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.324554 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7"] Oct 31 01:19:48.328128 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.328093 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7hl7l"] Oct 31 01:19:48.328285 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.328266 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452"] Oct 31 01:19:48.331512 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.331487 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8280d35c_b526_41e0_b4a2_0bd74b181789.slice/crio-f31ee3b89016e3c5ebe8b3e83fda821ba97cad1b04fa8d0e9a36bd972333d0e4 WatchSource:0}: Error finding container f31ee3b89016e3c5ebe8b3e83fda821ba97cad1b04fa8d0e9a36bd972333d0e4: Status 404 returned error can't find the container with id f31ee3b89016e3c5ebe8b3e83fda821ba97cad1b04fa8d0e9a36bd972333d0e4 Oct 31 01:19:48.332755 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.332730 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35a7382_ce08_44b1_8427_33a5d30252b7.slice/crio-002abe69fdf0cff71a45630f8257a0c4691f81e8abd9f91943f7f008d427a6ff WatchSource:0}: Error finding container 002abe69fdf0cff71a45630f8257a0c4691f81e8abd9f91943f7f008d427a6ff: Status 404 returned error can't find the container with id 002abe69fdf0cff71a45630f8257a0c4691f81e8abd9f91943f7f008d427a6ff Oct 31 01:19:48.359352 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.359327 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zkw9h"] Oct 31 01:19:48.361647 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.361621 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd04b736_45e6_44cc_a032_e47ef93641a9.slice/crio-2dbb82f197cb32d0f5f2b32c07629da6c7cbbca22a8aec120f53b54b91ba192a WatchSource:0}: Error finding container 2dbb82f197cb32d0f5f2b32c07629da6c7cbbca22a8aec120f53b54b91ba192a: Status 404 returned error can't find the container with id 2dbb82f197cb32d0f5f2b32c07629da6c7cbbca22a8aec120f53b54b91ba192a Oct 31 01:19:48.364350 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.364326 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-llbl2"] Oct 31 01:19:48.366362 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.366340 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fd2129_24b4_4c50_91be_bea78425e824.slice/crio-b5afaa159f5e08f6c39baed8b56a3bb5d973fecab7f63f2d0d87b39e60b8e627 WatchSource:0}: Error finding container b5afaa159f5e08f6c39baed8b56a3bb5d973fecab7f63f2d0d87b39e60b8e627: Status 404 returned error can't find the container with id b5afaa159f5e08f6c39baed8b56a3bb5d973fecab7f63f2d0d87b39e60b8e627 Oct 31 01:19:48.367208 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.367188 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj"] Oct 31 01:19:48.369181 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.368177 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jn8kf"] Oct 31 01:19:48.372908 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.372689 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda489a94c_56c7_45df_92cb_42177dadd9c1.slice/crio-c1869ac005b5383a2cb2e06cffcb88f61c1bc8adb4b3dd6555dd8235288817df WatchSource:0}: Error finding container c1869ac005b5383a2cb2e06cffcb88f61c1bc8adb4b3dd6555dd8235288817df: Status 404 returned error can't find the container with id c1869ac005b5383a2cb2e06cffcb88f61c1bc8adb4b3dd6555dd8235288817df Oct 31 01:19:48.372996 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:48.372927 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6f6573_29b9_4b30_a82c_2b6e8948583f.slice/crio-f5c80534eec72c823422f44570f612e5d30ee4655cbbcabfcb61cb77451518bc WatchSource:0}: Error finding container f5c80534eec72c823422f44570f612e5d30ee4655cbbcabfcb61cb77451518bc: Status 404 returned error can't find the container with id f5c80534eec72c823422f44570f612e5d30ee4655cbbcabfcb61cb77451518bc Oct 31 01:19:48.403216 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.403168 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:48.406130 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.406068 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:48.406431 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.406357 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 01:19:48.406431 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:48.406414 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls podName:df649d60-b1cd-408d-a0fc-e74fd86e6896 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:49.406397383 +0000 UTC m=+31.217326275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls") pod "dns-default-mrqzl" (UID: "df649d60-b1cd-408d-a0fc-e74fd86e6896") : secret "dns-default-metrics-tls" not found Oct 31 01:19:48.836267 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.836232 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" event={"ID":"f1b7733e-21c3-4770-bb58-d538802cbcaf","Type":"ContainerStarted","Data":"6f7a9d405c3c9516d18e65a68e75a342bca9dd60786e0c03cbc0cdc5b8995d28"} Oct 31 01:19:48.836696 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.836290 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" event={"ID":"f1b7733e-21c3-4770-bb58-d538802cbcaf","Type":"ContainerStarted","Data":"832394168dbe00bc7be0ea1b777b1528a2a81d660ac8fc4b55c020a9da8bed52"} Oct 31 01:19:48.837271 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.837251 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jn8kf" event={"ID":"a489a94c-56c7-45df-92cb-42177dadd9c1","Type":"ContainerStarted","Data":"c1869ac005b5383a2cb2e06cffcb88f61c1bc8adb4b3dd6555dd8235288817df"} Oct 31 01:19:48.838462 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.838443 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7hl7l" event={"ID":"8280d35c-b526-41e0-b4a2-0bd74b181789","Type":"ContainerStarted","Data":"f31ee3b89016e3c5ebe8b3e83fda821ba97cad1b04fa8d0e9a36bd972333d0e4"} Oct 31 01:19:48.839461 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.839443 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" event={"ID":"dd04b736-45e6-44cc-a032-e47ef93641a9","Type":"ContainerStarted","Data":"2dbb82f197cb32d0f5f2b32c07629da6c7cbbca22a8aec120f53b54b91ba192a"} Oct 31 01:19:48.840444 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.840420 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" event={"ID":"5e6f6573-29b9-4b30-a82c-2b6e8948583f","Type":"ContainerStarted","Data":"f5c80534eec72c823422f44570f612e5d30ee4655cbbcabfcb61cb77451518bc"} Oct 31 01:19:48.841737 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.841696 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-llbl2" event={"ID":"22fd2129-24b4-4c50-91be-bea78425e824","Type":"ContainerStarted","Data":"9192686f188ff870e5b3de441b55fc4661232c91d8109671ec947075923fd66a"} Oct 31 01:19:48.841810 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.841745 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-llbl2" event={"ID":"22fd2129-24b4-4c50-91be-bea78425e824","Type":"ContainerStarted","Data":"b5afaa159f5e08f6c39baed8b56a3bb5d973fecab7f63f2d0d87b39e60b8e627"} Oct 31 01:19:48.842875 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.842849 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" event={"ID":"3e5f8170-98cf-4487-84e2-0aeccb45ba6c","Type":"ContainerStarted","Data":"f69880c170d1166a954ac282d9b7cb1dade83cbf1a6ed9c6f6fb60b0857aa62f"} Oct 31 01:19:48.843801 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.843783 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" event={"ID":"8a3cb44a-89d0-4bb6-9815-3f52494b3047","Type":"ContainerStarted","Data":"10fd30ceae78ba7bb9f7a33c699cffea2273f788723cd2f013dc68a41ed9bba9"} Oct 31 01:19:48.844757 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.844742 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" event={"ID":"f35a7382-ce08-44b1-8427-33a5d30252b7","Type":"ContainerStarted","Data":"002abe69fdf0cff71a45630f8257a0c4691f81e8abd9f91943f7f008d427a6ff"} Oct 31 01:19:48.846047 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.846004 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" event={"ID":"57ac8842-181e-4186-b7b4-5175a511a0e6","Type":"ContainerStarted","Data":"7aed76deb743960b28b136ebea2f9251f615cd4fb2fe6578cbb0c96ce732ffe3"} Oct 31 01:19:48.846116 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.846065 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" event={"ID":"57ac8842-181e-4186-b7b4-5175a511a0e6","Type":"ContainerStarted","Data":"a36700057e998a5b42a124389eb2d08769bf66d4a0943caa30a279671e469959"} Oct 31 01:19:48.857746 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.857682 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" podStartSLOduration=411.85766631 podStartE2EDuration="6m51.85766631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:48.85651611 +0000 UTC m=+30.667445029" watchObservedRunningTime="2026-10-31 01:19:48.85766631 +0000 UTC m=+30.668595227" Oct 31 01:19:48.875977 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.875941 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-llbl2" podStartSLOduration=411.875929565 podStartE2EDuration="6m51.875929565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:48.875779401 +0000 UTC m=+30.686708311" watchObservedRunningTime="2026-10-31 01:19:48.875929565 +0000 UTC m=+30.686858478" Oct 31 01:19:48.894858 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:48.894825 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" podStartSLOduration=411.894814848 podStartE2EDuration="6m51.894814848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:48.894280552 +0000 UTC m=+30.705209465" watchObservedRunningTime="2026-10-31 01:19:48.894814848 +0000 UTC m=+30.705743760" Oct 31 01:19:49.315591 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.315543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:49.315591 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.315624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.315656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315675 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315774 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.315751917 +0000 UTC m=+33.126680828 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315785 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315787 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315826 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.315815499 +0000 UTC m=+33.126744394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315848 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.315833719 +0000 UTC m=+33.126762615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.315684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315881 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315889 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb9bcc569-v7h6k: secret "image-registry-tls" not found Oct 31 01:19:49.315926 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.315919 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls podName:bd51c16f-f327-4939-ad47-4acc6aaf62c2 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.315905806 +0000 UTC m=+33.126834698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls") pod "image-registry-fb9bcc569-v7h6k" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2") : secret "image-registry-tls" not found Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.315978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.316005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.316124 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.316110909 +0000 UTC m=+33.127039823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.316154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.316168 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.316206 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.316195333 +0000 UTC m=+33.127124230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.316314 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:49.316346 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.316339 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54d575cd44-5jzvn: secret "image-registry-tls" not found Oct 31 01:19:49.316650 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.316386 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls podName:b123e4c4-57cd-49f8-985b-158b4deab82d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.316372636 +0000 UTC m=+33.127301533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls") pod "image-registry-54d575cd44-5jzvn" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d") : secret "image-registry-tls" not found Oct 31 01:19:49.417637 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.417600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:49.417890 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.417784 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 01:19:49.417990 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:49.417936 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls podName:df649d60-b1cd-408d-a0fc-e74fd86e6896 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:51.417917236 +0000 UTC m=+33.228846144 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls") pod "dns-default-mrqzl" (UID: "df649d60-b1cd-408d-a0fc-e74fd86e6896") : secret "dns-default-metrics-tls" not found Oct 31 01:19:49.482417 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.482353 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:49.621114 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:49.621013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:50.537518 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:50.537385 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:50.860843 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:50.859865 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" event={"ID":"3e5f8170-98cf-4487-84e2-0aeccb45ba6c","Type":"ContainerStarted","Data":"b02af6ef37f9e28e9393f6956f503c150e841f8bca53a2d30e1efe9ab8d5179b"} Oct 31 01:19:50.882730 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:50.882656 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" podStartSLOduration=371.672504666 podStartE2EDuration="6m13.882638877s" totalImagesPullingTime="2.210134211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:50.879906472 +0000 UTC m=+32.690835386" watchObservedRunningTime="2026-10-31 01:19:50.882638877 +0000 UTC m=+32.693567791" Oct 31 01:19:51.034498 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.034460 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:51.176236 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.176106 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:51.337006 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.336967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:51.337212 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:19:51.337212 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:51.337212 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:51.337212 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:51.337443 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:51.337443 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337296 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:51.337443 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:51.337443 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.337348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:51.337643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337471 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:19:51.337643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337524 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:51.337643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337537 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.337518391 +0000 UTC m=+37.148447281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:19:51.337643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337544 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54d575cd44-5jzvn: secret "image-registry-tls" not found Oct 31 01:19:51.337643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337582 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:19:51.337643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337600 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls podName:b123e4c4-57cd-49f8-985b-158b4deab82d nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.337580836 +0000 UTC m=+37.148509750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls") pod "image-registry-54d575cd44-5jzvn" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d") : secret "image-registry-tls" not found Oct 31 01:19:51.337643 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337616 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.33760628 +0000 UTC m=+37.148535184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337660 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337663 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337691 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs podName:43b11ef2-886e-4d57-9700-089f51fa360e nodeName:}" failed. No retries permitted until 2026-10-31 01:20:23.33768249 +0000 UTC m=+65.148611381 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs") pod "network-metrics-daemon-wff52" (UID: "43b11ef2-886e-4d57-9700-089f51fa360e") : secret "metrics-daemon-secret" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337705 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.337697304 +0000 UTC m=+37.148626201 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337767 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337777 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb9bcc569-v7h6k: secret "image-registry-tls" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337785 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.337774331 +0000 UTC m=+37.148703225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337805 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls podName:bd51c16f-f327-4939-ad47-4acc6aaf62c2 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.337795188 +0000 UTC m=+37.148724094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls") pod "image-registry-fb9bcc569-v7h6k" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2") : secret "image-registry-tls" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337835 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:51.338064 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.337866 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.33785708 +0000 UTC m=+37.148785974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:51.342201 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.342175 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:51.438816 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.438080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:51.438816 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.438175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:51.439285 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.438347 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 01:19:51.439360 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:51.439347 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls podName:df649d60-b1cd-408d-a0fc-e74fd86e6896 nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.439322086 +0000 UTC m=+37.250250977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls") pod "dns-default-mrqzl" (UID: "df649d60-b1cd-408d-a0fc-e74fd86e6896") : secret "dns-default-metrics-tls" not found Oct 31 01:19:51.446578 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.446512 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:51.613875 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.613835 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:51.628665 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.628634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:19:51.907654 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.907350 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-klvct"] Oct 31 01:19:51.910427 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:51.910399 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578616cb_95f4_49ef_afec_65f65cdb2f71.slice/crio-6d668c7e8a5b00abfdb515959e6d0a76472a63e6663a7e10bb03d88043fc23a4 WatchSource:0}: Error finding container 6d668c7e8a5b00abfdb515959e6d0a76472a63e6663a7e10bb03d88043fc23a4: Status 404 returned error can't find the container with id 6d668c7e8a5b00abfdb515959e6d0a76472a63e6663a7e10bb03d88043fc23a4 Oct 31 01:19:51.943022 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:51.942991 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c5d98"] Oct 31 01:19:51.948260 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:51.948232 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e587446_856d_4580_a69f_94cd5953a81d.slice/crio-0647a8d1532f7ed070bcac68f46da74acef76e2ac56ed35eec91aa94f9cdf2a3 WatchSource:0}: Error finding container 0647a8d1532f7ed070bcac68f46da74acef76e2ac56ed35eec91aa94f9cdf2a3: Status 404 returned error can't find the container with id 0647a8d1532f7ed070bcac68f46da74acef76e2ac56ed35eec91aa94f9cdf2a3 Oct 31 01:19:52.868537 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:52.868447 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c5d98" event={"ID":"7e587446-856d-4580-a69f-94cd5953a81d","Type":"ContainerStarted","Data":"0647a8d1532f7ed070bcac68f46da74acef76e2ac56ed35eec91aa94f9cdf2a3"} Oct 31 01:19:52.870268 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:52.870237 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-klvct" event={"ID":"578616cb-95f4-49ef-afec-65f65cdb2f71","Type":"ContainerStarted","Data":"6d668c7e8a5b00abfdb515959e6d0a76472a63e6663a7e10bb03d88043fc23a4"} Oct 31 01:19:54.780872 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.780827 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8jv29"] Oct 31 01:19:54.804808 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.803020 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-slnps-lhfzm"] Oct 31 01:19:54.804808 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.804220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.812436 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.809595 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 01:19:54.812436 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.810347 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bztmp\"" Oct 31 01:19:54.812436 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.810402 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 01:19:54.819018 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.818998 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jv29"] Oct 31 01:19:54.819121 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.819048 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-slnps-lhfzm"] Oct 31 01:19:54.819186 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.819135 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:54.832469 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.832325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:54.880048 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.880012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-data-volume\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.880217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.880062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-crio-socket\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.880217 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.880140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.880329 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.880310 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.880400 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.880342 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-api-access-ndrdz\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.881328 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.881291 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" event={"ID":"8a3cb44a-89d0-4bb6-9815-3f52494b3047","Type":"ContainerStarted","Data":"89f8d684f504ecc87d8b032fc2912a6406a7377c86a2224f4196f88d958a9be3"} Oct 31 01:19:54.884258 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.883868 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" event={"ID":"f35a7382-ce08-44b1-8427-33a5d30252b7","Type":"ContainerStarted","Data":"72160e323b3c04c7aed02102262f85a6c6e1a950632c2f92588e55b577874275"} Oct 31 01:19:54.887170 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.887143 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jn8kf" event={"ID":"a489a94c-56c7-45df-92cb-42177dadd9c1","Type":"ContainerStarted","Data":"8c076ae4948b7eca061b9524f5cb09bc27eb9caa1054478d1d7bdfce118345d0"} Oct 31 01:19:54.889850 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.889477 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7hl7l" event={"ID":"8280d35c-b526-41e0-b4a2-0bd74b181789","Type":"ContainerStarted","Data":"ab058ea00b533dbe69a0edec4719238d620cb771d874d2b2f2374f747aa7703f"} Oct 31 01:19:54.892685 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.892664 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/0.log" Oct 31 01:19:54.892807 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.892702 2575 generic.go:356] "Generic (PLEG): container finished" podID="dd04b736-45e6-44cc-a032-e47ef93641a9" containerID="cc966369420d70477881c843b3ad1ffb8b1852c98fade9d406fb6e2f613e6838" exitCode=255 Oct 31 01:19:54.892807 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.892760 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" event={"ID":"dd04b736-45e6-44cc-a032-e47ef93641a9","Type":"ContainerDied","Data":"cc966369420d70477881c843b3ad1ffb8b1852c98fade9d406fb6e2f613e6838"} Oct 31 01:19:54.893050 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.893033 2575 scope.go:122] "RemoveContainer" containerID="cc966369420d70477881c843b3ad1ffb8b1852c98fade9d406fb6e2f613e6838" Oct 31 01:19:54.905750 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.905333 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" podStartSLOduration=372.241137494 podStartE2EDuration="6m17.905319047s" totalImagesPullingTime="5.664181553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:54.90201107 +0000 UTC m=+36.712939984" watchObservedRunningTime="2026-10-31 01:19:54.905319047 +0000 UTC m=+36.716247961" Oct 31 01:19:54.924149 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.924110 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-7hl7l" podStartSLOduration=372.274497118 podStartE2EDuration="6m17.924097578s" totalImagesPullingTime="5.64960046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:54.922923918 +0000 UTC m=+36.733852833" watchObservedRunningTime="2026-10-31 01:19:54.924097578 +0000 UTC m=+36.735026491" Oct 31 01:19:54.971131 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.971080 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" podStartSLOduration=372.333358662 podStartE2EDuration="6m17.97106016s" totalImagesPullingTime="5.637701498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:54.970192844 +0000 UTC m=+36.781121759" watchObservedRunningTime="2026-10-31 01:19:54.97106016 +0000 UTC m=+36.781989076" Oct 31 01:19:54.982085 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.982223 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:54.982223 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-api-access-ndrdz\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.982331 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:54.982389 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:54.982659 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-data-volume\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.982659 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-crio-socket\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.982845 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982699 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbbj\" (UniqueName: \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:54.982845 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.982759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.982915 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:54.982901 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:19:54.983151 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:54.983132 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.482952811 +0000 UTC m=+37.293881703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:19:54.986542 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:54.986522 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 01:19:54.986646 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:54.986581 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:19:55.486565008 +0000 UTC m=+37.297493913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : configmap "kube-rbac-proxy" not found Oct 31 01:19:54.988001 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.987977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-data-volume\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.988189 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.988166 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-crio-socket\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:54.990138 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:54.990111 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:55.003688 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.003670 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-api-access-ndrdz\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:55.008093 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.008052 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jn8kf" podStartSLOduration=2.410516613 podStartE2EDuration="8.008040425s" totalImagesPullingTime="5.597523812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:55.004452956 +0000 UTC m=+36.815381871" watchObservedRunningTime="2026-10-31 01:19:55.008040425 +0000 UTC m=+36.818969338" Oct 31 01:19:55.084743 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.084668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbbj\" (UniqueName: \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.084890 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.084854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.084953 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.084910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.084953 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.084947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.085356 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.085331 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.085625 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.085604 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.086271 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.086236 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.106160 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.106133 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbbj\" (UniqueName: \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.126117 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.125049 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2"] Oct 31 01:19:55.150339 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.150305 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77"] Oct 31 01:19:55.150923 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.150903 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" Oct 31 01:19:55.164398 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.164374 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2"] Oct 31 01:19:55.164528 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.164412 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77"] Oct 31 01:19:55.164528 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.164483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" Oct 31 01:19:55.179956 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.179772 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:19:55.184888 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.184859 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:55.196560 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.196531 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 01:19:55.196827 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.196809 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 01:19:55.225641 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.225583 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4ls7t\"" Oct 31 01:19:55.288056 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.287782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtx7\" (UniqueName: \"kubernetes.io/projected/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6-kube-api-access-rdtx7\") pod \"migrator-66df5d496b-8tkh2\" (UID: \"6ae589ef-bc72-4b28-a05a-14dc01aa1dd6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" Oct 31 01:19:55.288056 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.287937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tprw\" (UniqueName: \"kubernetes.io/projected/51211307-a4ca-4d06-95f7-02a7e5fef197-kube-api-access-7tprw\") pod \"migrator-66df5d496b-6km77\" (UID: \"51211307-a4ca-4d06-95f7-02a7e5fef197\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" Oct 31 01:19:55.327887 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.327749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:55.388816 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.388777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:55.388816 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.388813 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.388844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tprw\" (UniqueName: \"kubernetes.io/projected/51211307-a4ca-4d06-95f7-02a7e5fef197-kube-api-access-7tprw\") pod \"migrator-66df5d496b-6km77\" (UID: \"51211307-a4ca-4d06-95f7-02a7e5fef197\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.388866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.388917 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.388949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.388983 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.388955873 +0000 UTC m=+45.199884783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.389015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtx7\" (UniqueName: \"kubernetes.io/projected/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6-kube-api-access-rdtx7\") pod \"migrator-66df5d496b-8tkh2\" (UID: \"6ae589ef-bc72-4b28-a05a-14dc01aa1dd6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389035 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.389051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389073 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.389062572 +0000 UTC m=+45.199991465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:19:55.389104 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.389087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389142 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389156 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fb9bcc569-v7h6k: secret "image-registry-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389187 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389195 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls podName:bd51c16f-f327-4939-ad47-4acc6aaf62c2 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.389181423 +0000 UTC m=+45.200110328 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls") pod "image-registry-fb9bcc569-v7h6k" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2") : secret "image-registry-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389157 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389209 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.389202467 +0000 UTC m=+45.200131358 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389230 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.389220986 +0000 UTC m=+45.200149897 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389455 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389467 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54d575cd44-5jzvn: secret "image-registry-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389500 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls podName:b123e4c4-57cd-49f8-985b-158b4deab82d nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.389488358 +0000 UTC m=+45.200417267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls") pod "image-registry-54d575cd44-5jzvn" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d") : secret "image-registry-tls" not found Oct 31 01:19:55.389567 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389553 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:19:55.390005 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.389586 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.38957523 +0000 UTC m=+45.200504121 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:19:55.403672 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.403645 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tprw\" (UniqueName: \"kubernetes.io/projected/51211307-a4ca-4d06-95f7-02a7e5fef197-kube-api-access-7tprw\") pod \"migrator-66df5d496b-6km77\" (UID: \"51211307-a4ca-4d06-95f7-02a7e5fef197\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" Oct 31 01:19:55.405091 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.405069 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtx7\" (UniqueName: \"kubernetes.io/projected/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6-kube-api-access-rdtx7\") pod \"migrator-66df5d496b-8tkh2\" (UID: \"6ae589ef-bc72-4b28-a05a-14dc01aa1dd6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" Oct 31 01:19:55.481051 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.481018 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" Oct 31 01:19:55.489589 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.489554 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:55.489735 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.489673 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 01:19:55.489791 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.489706 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:55.489791 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.489759 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:19:56.489742484 +0000 UTC m=+38.300671390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : configmap "kube-rbac-proxy" not found Oct 31 01:19:55.489915 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.489810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:19:55.489915 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.489832 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:19:55.489915 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.489894 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:19:56.48987529 +0000 UTC m=+38.300804196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:19:55.490082 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.489937 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 01:19:55.490082 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.489964 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls podName:df649d60-b1cd-408d-a0fc-e74fd86e6896 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:03.489956253 +0000 UTC m=+45.300885192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls") pod "dns-default-mrqzl" (UID: "df649d60-b1cd-408d-a0fc-e74fd86e6896") : secret "dns-default-metrics-tls" not found Oct 31 01:19:55.503064 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.502886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" Oct 31 01:19:55.507461 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.507050 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-slnps-lhfzm"] Oct 31 01:19:55.510255 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:55.510171 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e634f3d_0490_480a_9484_a1ee415c5fb1.slice/crio-f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d WatchSource:0}: Error finding container f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d: Status 404 returned error can't find the container with id f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d Oct 31 01:19:55.518750 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.518389 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:55.558885 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.558205 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-bw6g8"] Oct 31 01:19:55.583973 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.583903 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-bw6g8"] Oct 31 01:19:55.584129 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.584048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.589760 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.589736 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 01:19:55.589884 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.589856 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dnq6d\"" Oct 31 01:19:55.589957 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.589924 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 01:19:55.590036 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.590010 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 01:19:55.590400 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.590366 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 01:19:55.691415 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.691370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvkm\" (UniqueName: \"kubernetes.io/projected/5472b622-19ab-4582-a5ef-4f09314f0213-kube-api-access-rjvkm\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.691600 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.691443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5472b622-19ab-4582-a5ef-4f09314f0213-signing-key\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.691600 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.691493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5472b622-19ab-4582-a5ef-4f09314f0213-signing-cabundle\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.785563 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.785530 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2"] Oct 31 01:19:55.789617 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:55.789576 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae589ef_bc72_4b28_a05a_14dc01aa1dd6.slice/crio-fcc542ccc673cc228cb713f90fdb5af27c51aa0c14b88f4858356a429e5d0ddd WatchSource:0}: Error finding container fcc542ccc673cc228cb713f90fdb5af27c51aa0c14b88f4858356a429e5d0ddd: Status 404 returned error can't find the container with id fcc542ccc673cc228cb713f90fdb5af27c51aa0c14b88f4858356a429e5d0ddd Oct 31 01:19:55.793883 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.792221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvkm\" (UniqueName: \"kubernetes.io/projected/5472b622-19ab-4582-a5ef-4f09314f0213-kube-api-access-rjvkm\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.793883 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.792257 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5472b622-19ab-4582-a5ef-4f09314f0213-signing-key\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.793883 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.792285 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5472b622-19ab-4582-a5ef-4f09314f0213-signing-cabundle\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.794482 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.794399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5472b622-19ab-4582-a5ef-4f09314f0213-signing-cabundle\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.796194 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.796171 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5472b622-19ab-4582-a5ef-4f09314f0213-signing-key\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.809232 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.809215 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvkm\" (UniqueName: \"kubernetes.io/projected/5472b622-19ab-4582-a5ef-4f09314f0213-kube-api-access-rjvkm\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.842529 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.842457 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77"] Oct 31 01:19:55.848776 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:55.848744 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51211307_a4ca_4d06_95f7_02a7e5fef197.slice/crio-ec19dd605f3028be1173c5ab62fc678af447d2a272e168ef8aa9d4fb0b7ae5c5 WatchSource:0}: Error finding container ec19dd605f3028be1173c5ab62fc678af447d2a272e168ef8aa9d4fb0b7ae5c5: Status 404 returned error can't find the container with id ec19dd605f3028be1173c5ab62fc678af447d2a272e168ef8aa9d4fb0b7ae5c5 Oct 31 01:19:55.898777 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.898737 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slnps-lhfzm" event={"ID":"9e634f3d-0490-480a-9484-a1ee415c5fb1","Type":"ContainerStarted","Data":"06a2c9cfe6606ac01e54ec86918d53b44968543e1f8af1dd095348dff99b3108"} Oct 31 01:19:55.898917 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.898804 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slnps-lhfzm" event={"ID":"9e634f3d-0490-480a-9484-a1ee415c5fb1","Type":"ContainerStarted","Data":"f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d"} Oct 31 01:19:55.900452 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.900425 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" event={"ID":"51211307-a4ca-4d06-95f7-02a7e5fef197","Type":"ContainerStarted","Data":"ec19dd605f3028be1173c5ab62fc678af447d2a272e168ef8aa9d4fb0b7ae5c5"} Oct 31 01:19:55.903227 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.903207 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:19:55.903684 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.903661 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/0.log" Oct 31 01:19:55.903817 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.903698 2575 generic.go:356] "Generic (PLEG): container finished" podID="dd04b736-45e6-44cc-a032-e47ef93641a9" containerID="176f3b933e5201fea082acd69c7789423a6693b3fe23407901e56699c4c6cc14" exitCode=255 Oct 31 01:19:55.903817 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.903777 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" event={"ID":"dd04b736-45e6-44cc-a032-e47ef93641a9","Type":"ContainerDied","Data":"176f3b933e5201fea082acd69c7789423a6693b3fe23407901e56699c4c6cc14"} Oct 31 01:19:55.903902 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.903820 2575 scope.go:122] "RemoveContainer" containerID="cc966369420d70477881c843b3ad1ffb8b1852c98fade9d406fb6e2f613e6838" Oct 31 01:19:55.904148 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.904132 2575 scope.go:122] "RemoveContainer" containerID="176f3b933e5201fea082acd69c7789423a6693b3fe23407901e56699c4c6cc14" Oct 31 01:19:55.904333 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:55.904311 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zkw9h_openshift-console-operator(dd04b736-45e6-44cc-a032-e47ef93641a9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" podUID="dd04b736-45e6-44cc-a032-e47ef93641a9" Oct 31 01:19:55.905825 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.905641 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" event={"ID":"6ae589ef-bc72-4b28-a05a-14dc01aa1dd6","Type":"ContainerStarted","Data":"fcc542ccc673cc228cb713f90fdb5af27c51aa0c14b88f4858356a429e5d0ddd"} Oct 31 01:19:55.917049 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.917029 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:19:55.921224 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.921177 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-slnps-lhfzm" podStartSLOduration=1.921159542 podStartE2EDuration="1.921159542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:55.92007205 +0000 UTC m=+37.731000963" watchObservedRunningTime="2026-10-31 01:19:55.921159542 +0000 UTC m=+37.732088456" Oct 31 01:19:55.961874 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:55.961677 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:56.105567 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.105490 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:56.261446 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.261414 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-bw6g8"] Oct 31 01:19:56.278657 ip-10-0-141-145 kubenswrapper[2575]: W1031 01:19:56.278628 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5472b622_19ab_4582_a5ef_4f09314f0213.slice/crio-9a1535e14a14cc97ff06a79a4502de33888575aeaa2baf6076a37e1b19f32181 WatchSource:0}: Error finding container 9a1535e14a14cc97ff06a79a4502de33888575aeaa2baf6076a37e1b19f32181: Status 404 returned error can't find the container with id 9a1535e14a14cc97ff06a79a4502de33888575aeaa2baf6076a37e1b19f32181 Oct 31 01:19:56.500524 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.500484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:56.500687 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.500647 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:56.501330 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:56.500856 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 01:19:56.501330 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:56.500934 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:19:58.500911866 +0000 UTC m=+40.311840760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : configmap "kube-rbac-proxy" not found Oct 31 01:19:56.501330 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:56.500858 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:19:56.501330 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:56.501298 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:19:58.501284062 +0000 UTC m=+40.312212957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:19:56.792505 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.792213 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:56.912227 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.912192 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:19:56.912615 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.912599 2575 scope.go:122] "RemoveContainer" containerID="176f3b933e5201fea082acd69c7789423a6693b3fe23407901e56699c4c6cc14" Oct 31 01:19:56.912877 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:56.912845 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zkw9h_openshift-console-operator(dd04b736-45e6-44cc-a032-e47ef93641a9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" podUID="dd04b736-45e6-44cc-a032-e47ef93641a9" Oct 31 01:19:56.916420 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.916395 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" event={"ID":"5e6f6573-29b9-4b30-a82c-2b6e8948583f","Type":"ContainerStarted","Data":"83b9177675c914b3fc5c493596a93f135f5eb8f36e434d4b0043f963a78c2550"} Oct 31 01:19:56.920132 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.920110 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" event={"ID":"5472b622-19ab-4582-a5ef-4f09314f0213","Type":"ContainerStarted","Data":"361b27667fcb76cd7f44fb8316988ab0cf1bbb5c175735829bac2e380fa139d4"} Oct 31 01:19:56.920231 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.920151 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" event={"ID":"5472b622-19ab-4582-a5ef-4f09314f0213","Type":"ContainerStarted","Data":"9a1535e14a14cc97ff06a79a4502de33888575aeaa2baf6076a37e1b19f32181"} Oct 31 01:19:56.962069 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.961564 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" podStartSLOduration=372.028236175 podStartE2EDuration="6m19.961545741s" totalImagesPullingTime="7.933309566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:56.95987019 +0000 UTC m=+38.770799104" watchObservedRunningTime="2026-10-31 01:19:56.961545741 +0000 UTC m=+38.772474655" Oct 31 01:19:56.980810 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:56.980356 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" podStartSLOduration=1.9803453819999999 podStartE2EDuration="1.980345382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:56.979567045 +0000 UTC m=+38.790495957" watchObservedRunningTime="2026-10-31 01:19:56.980345382 +0000 UTC m=+38.791274294" Oct 31 01:19:57.271290 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.271231 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:57.415219 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.415113 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:57.552491 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.552341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:57.763912 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.763880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:57.939313 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.939003 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:57.977581 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.977547 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:57.977688 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.977608 2575 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:19:57.978026 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:57.978011 2575 scope.go:122] "RemoveContainer" containerID="176f3b933e5201fea082acd69c7789423a6693b3fe23407901e56699c4c6cc14" Oct 31 01:19:57.978251 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:57.978229 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zkw9h_openshift-console-operator(dd04b736-45e6-44cc-a032-e47ef93641a9)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" podUID="dd04b736-45e6-44cc-a032-e47ef93641a9" Oct 31 01:19:58.264155 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.264126 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:58.523708 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.523627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:58.523862 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.523781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:19:58.523862 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:58.523816 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:19:58.523997 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:58.523895 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.523874851 +0000 UTC m=+44.334803758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:19:58.523997 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:58.523896 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 01:19:58.523997 ip-10-0-141-145 kubenswrapper[2575]: E1031 01:19:58.523948 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.523932011 +0000 UTC m=+44.334860917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : configmap "kube-rbac-proxy" not found Oct 31 01:19:58.750905 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.750868 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:58.927938 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.927865 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" event={"ID":"51211307-a4ca-4d06-95f7-02a7e5fef197","Type":"ContainerStarted","Data":"6110e507ae1095ec5ecbc2ff3c2611eb9e628a64959a2c23b6fb284d2d027922"} Oct 31 01:19:58.929487 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.929458 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-klvct" event={"ID":"578616cb-95f4-49ef-afec-65f65cdb2f71","Type":"ContainerStarted","Data":"ccff7738efd76546677fcef9f194c022d1b7a9ffcd826769ab1736c5c8b2fd69"} Oct 31 01:19:58.929611 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.929514 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:19:58.931038 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.931018 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" event={"ID":"6ae589ef-bc72-4b28-a05a-14dc01aa1dd6","Type":"ContainerStarted","Data":"5241fd98a61523c66cb1bc4b396600b90fcf4b41ea609a76ee54a9240ac34c81"} Oct 31 01:19:58.947958 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.947919 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" podStartSLOduration=1.468464394 podStartE2EDuration="3.947907746s" totalImagesPullingTime="2.479443352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:58.946754659 +0000 UTC m=+40.757683573" watchObservedRunningTime="2026-10-31 01:19:58.947907746 +0000 UTC m=+40.758836660" Oct 31 01:19:58.966690 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.966652 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" podStartSLOduration=1.431555811 podStartE2EDuration="3.966640542s" totalImagesPullingTime="2.535084731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:58.965016995 +0000 UTC m=+40.775945909" watchObservedRunningTime="2026-10-31 01:19:58.966640542 +0000 UTC m=+40.777569454" Oct 31 01:19:58.984556 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:58.984522 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-klvct" podStartSLOduration=33.992536945 podStartE2EDuration="39.984509837s" totalImagesPullingTime="5.991972892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:19:58.983780482 +0000 UTC m=+40.794709395" watchObservedRunningTime="2026-10-31 01:19:58.984509837 +0000 UTC m=+40.795438753" Oct 31 01:19:59.126136 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:59.126105 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:59.330033 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:59.330007 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:19:59.527859 ip-10-0-141-145 kubenswrapper[2575]: I1031 01:19:59.527789 2575 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 01:19:59.527931 ip-10-0-141-145 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 01:19:59.535344 ip-10-0-141-145 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 01:19:59.535510 ip-10-0-141-145 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 01:19:59.535557 ip-10-0-141-145 systemd[1]: kubelet.service: Consumed 3.163s CPU time. Oct 31 01:19:59.546233 ip-10-0-141-145 systemd[1]: Starting Kubernetes Kubelet... Oct 31 01:19:59.650102 ip-10-0-141-145 kubenswrapper[5090]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 01:19:59.650102 ip-10-0-141-145 kubenswrapper[5090]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 01:19:59.650102 ip-10-0-141-145 kubenswrapper[5090]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 01:19:59.650102 ip-10-0-141-145 kubenswrapper[5090]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 01:19:59.653051 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.652923 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.653051 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653047 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.653051 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653053 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653058 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653069 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653073 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653077 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653083 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653088 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653093 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653098 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653102 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653107 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653112 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653117 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653121 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653162 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653167 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653171 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653175 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653180 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653184 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.653275 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653188 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653192 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653196 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653200 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653205 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653209 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653213 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653223 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653228 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653232 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653237 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653244 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653250 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653255 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653260 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653265 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653269 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653273 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653278 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.654129 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653288 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653292 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653296 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653300 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653305 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653309 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653313 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653317 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653349 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653896 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653917 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653923 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653926 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653930 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653951 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653955 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653958 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653961 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.653964 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.654626 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654095 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654115 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654119 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654128 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654131 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654135 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654137 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654140 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654143 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654147 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654150 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654152 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654155 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654158 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654162 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654168 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654172 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654175 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654178 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654182 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.655113 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654185 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654188 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654191 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654193 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654196 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654199 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654201 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654204 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654206 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654210 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654213 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654216 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654219 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654221 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654224 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654227 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654230 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654232 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654237 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654240 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.655639 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654242 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654245 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654248 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654250 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654253 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.654255 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655109 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655116 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655120 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655124 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655127 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655130 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655133 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655136 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655139 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655141 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655145 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655148 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655150 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655153 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.656132 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655156 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655159 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655162 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655164 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655167 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655170 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655172 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655176 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655180 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655182 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655185 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655188 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655190 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655193 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655195 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655198 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655201 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655204 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655208 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.656613 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655211 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655213 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655216 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655219 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655221 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655224 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655226 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655229 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655232 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655235 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655238 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655242 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655244 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655247 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655250 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655252 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655255 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655258 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655261 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655263 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.657131 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655266 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655268 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655271 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655273 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655276 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655280 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655284 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655288 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655291 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655293 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655296 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655299 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655302 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655304 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655307 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655309 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655312 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655315 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655318 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.657604 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655321 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655323 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655326 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655329 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655332 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655335 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655337 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655340 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655342 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655345 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655347 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655350 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655352 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655355 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655357 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655360 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655363 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655365 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655368 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655370 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.658088 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655373 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655376 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655378 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655381 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655384 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655387 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655390 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655393 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655395 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655398 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655400 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655403 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655407 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655409 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655988 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.655996 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656000 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656003 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656005 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656009 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.658585 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656014 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656017 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656020 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656023 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656025 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656028 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656031 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656034 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656037 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656039 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656042 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656044 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656047 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656050 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656053 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656056 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656059 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656062 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656064 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.659075 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656067 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656070 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656072 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656075 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656077 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656080 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656083 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656085 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656088 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656090 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656094 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656096 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656099 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656102 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656104 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656106 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656109 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656111 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656114 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.659525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656117 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656120 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656122 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656125 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656128 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656130 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656134 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656137 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656140 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656142 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656145 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656148 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656150 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656153 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656155 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656158 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656161 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656163 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656166 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656168 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.660010 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656171 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656173 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656176 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656178 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656182 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656184 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656187 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656190 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656192 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656195 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656197 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656200 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656203 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656205 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656208 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656210 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656213 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656216 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656218 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656221 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.660838 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656223 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656226 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656228 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656232 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656236 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656239 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656242 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656245 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656247 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656250 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656252 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656256 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656258 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656261 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656264 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656267 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656269 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656272 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656275 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.661698 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656278 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656280 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.656283 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656382 5090 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656394 5090 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656401 5090 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656409 5090 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656414 5090 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656417 5090 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656421 5090 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656426 5090 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656429 5090 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656432 5090 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656436 5090 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656439 5090 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656442 5090 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656445 5090 flags.go:64] FLAG: --cgroup-root="" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656448 5090 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656451 5090 flags.go:64] FLAG: --client-ca-file="" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656454 5090 flags.go:64] FLAG: --cloud-provider="external" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656458 5090 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656462 5090 flags.go:64] FLAG: --cluster-domain="" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656465 5090 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 01:19:59.662538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656468 5090 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656471 5090 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656475 5090 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656485 5090 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656488 5090 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656491 5090 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656494 5090 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656497 5090 flags.go:64] FLAG: --contention-profiling="false" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656501 5090 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656504 5090 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656507 5090 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656511 5090 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656516 5090 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656519 5090 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656522 5090 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656526 5090 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656529 5090 flags.go:64] FLAG: --enable-server="true" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656532 5090 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656537 5090 flags.go:64] FLAG: --event-burst="100" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656540 5090 flags.go:64] FLAG: --event-qps="50" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656543 5090 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656547 5090 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656550 5090 flags.go:64] FLAG: --eviction-hard="" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656554 5090 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 01:19:59.663538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656557 5090 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656560 5090 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656563 5090 flags.go:64] FLAG: --eviction-soft="" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656567 5090 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656569 5090 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656572 5090 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656576 5090 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656578 5090 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656582 5090 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656585 5090 flags.go:64] FLAG: --feature-gates="" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656589 5090 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656595 5090 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656598 5090 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656602 5090 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656605 5090 flags.go:64] FLAG: --healthz-port="10248" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656608 5090 flags.go:64] FLAG: --help="false" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656611 5090 flags.go:64] FLAG: --hostname-override="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656616 5090 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656619 5090 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656622 5090 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656626 5090 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656629 5090 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656632 5090 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 01:19:59.664653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656636 5090 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656639 5090 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656642 5090 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656645 5090 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656648 5090 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656651 5090 flags.go:64] FLAG: --kube-reserved="" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656654 5090 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656657 5090 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656660 5090 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656663 5090 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656665 5090 flags.go:64] FLAG: --lock-file="" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656668 5090 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656671 5090 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656674 5090 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656680 5090 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656683 5090 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656686 5090 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656689 5090 flags.go:64] FLAG: --logging-format="text" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656692 5090 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656696 5090 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656700 5090 flags.go:64] FLAG: --manifest-url="" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656702 5090 flags.go:64] FLAG: --manifest-url-header="" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656707 5090 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656726 5090 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656732 5090 flags.go:64] FLAG: --max-pods="110" Oct 31 01:19:59.665653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656735 5090 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656738 5090 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656743 5090 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656746 5090 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656749 5090 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656752 5090 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656755 5090 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656763 5090 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656766 5090 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656769 5090 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656772 5090 flags.go:64] FLAG: --pod-cidr="" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656775 5090 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656778 5090 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656781 5090 flags.go:64] FLAG: --pods-per-core="0" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656784 5090 flags.go:64] FLAG: --port="10250" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656787 5090 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656790 5090 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-085afaf3362abe062" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656793 5090 flags.go:64] FLAG: --qos-reserved="" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656796 5090 flags.go:64] FLAG: --read-only-port="10255" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656799 5090 flags.go:64] FLAG: --register-node="true" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656802 5090 flags.go:64] FLAG: --register-with-taints="" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656806 5090 flags.go:64] FLAG: --registry-burst="10" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656809 5090 flags.go:64] FLAG: --registry-qps="5" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656813 5090 flags.go:64] FLAG: --reserved-cpus="" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656816 5090 flags.go:64] FLAG: --reserved-memory="" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656820 5090 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 01:19:59.666760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656822 5090 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656827 5090 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656830 5090 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656833 5090 flags.go:64] FLAG: --runonce="false" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656836 5090 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656839 5090 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656842 5090 flags.go:64] FLAG: --seccomp-default="false" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656845 5090 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656849 5090 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656853 5090 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656856 5090 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656859 5090 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656862 5090 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656865 5090 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656868 5090 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656871 5090 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656874 5090 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656877 5090 flags.go:64] FLAG: --system-cgroups="" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656880 5090 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656885 5090 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656888 5090 flags.go:64] FLAG: --tls-cert-file="" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656891 5090 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656895 5090 flags.go:64] FLAG: --tls-min-version="" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656898 5090 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656901 5090 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 01:19:59.667934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656903 5090 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656906 5090 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656909 5090 flags.go:64] FLAG: --v="2" Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656914 5090 flags.go:64] FLAG: --version="false" Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656918 5090 flags.go:64] FLAG: --vmodule="" Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656922 5090 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.656925 5090 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657074 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657080 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657084 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657086 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657090 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657093 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657095 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657098 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657101 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657103 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657107 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657109 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657112 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657115 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.668996 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657117 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657120 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657123 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657126 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657128 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657131 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657133 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657136 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657139 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657142 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657145 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657147 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657149 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657152 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657155 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657159 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657162 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657164 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657167 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.669888 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657170 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657173 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657175 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657178 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657181 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657183 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657186 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657188 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657191 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657193 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657196 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657199 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657203 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657207 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657210 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657213 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657216 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657218 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657221 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.670458 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657223 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657226 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657229 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657231 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657234 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657236 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657239 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657242 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657244 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657248 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657251 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657254 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657257 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657259 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657262 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657264 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657267 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657269 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657272 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657274 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.670993 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657277 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657280 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657282 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657285 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657288 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657291 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657294 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657296 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657299 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657301 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657304 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657306 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657309 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657312 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657314 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657317 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657320 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657322 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657325 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657327 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.671524 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657330 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657334 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657337 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657339 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657342 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657344 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657347 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657349 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657352 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657356 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657360 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657362 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657365 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657368 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.657393 5090 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 01:19:59.672122 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657430 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657434 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657438 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657440 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657443 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657446 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657449 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657451 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657454 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657456 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657459 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657462 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657465 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657468 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657470 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657473 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657475 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657478 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657482 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657485 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.672526 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657487 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657490 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657493 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657496 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657499 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657502 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657506 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657509 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657512 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657514 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657517 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657520 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657523 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657525 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657528 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657531 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657534 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657537 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657539 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657542 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.673282 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657545 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657547 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657550 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657553 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657555 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657558 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657561 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657564 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657566 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657569 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657574 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657576 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657579 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657581 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657584 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657588 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657590 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657593 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657596 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.673931 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657599 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657601 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657604 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657607 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657610 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657613 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657616 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657618 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657621 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657625 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657627 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657630 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657632 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657635 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657637 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657640 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657642 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657646 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657649 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.674435 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657652 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657655 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657657 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657660 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657664 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657667 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657669 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657672 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657675 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657679 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657681 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657684 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657687 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657690 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657693 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657696 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657698 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657701 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657704 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657706 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657709 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.674985 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657732 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657735 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657739 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657742 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657744 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657747 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.657750 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.663119 5090 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.663136 5090 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.663161 5090 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.663168 5090 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663256 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663265 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663270 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663275 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663280 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663285 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663290 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663294 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663299 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.675790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663304 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663308 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663312 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663316 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663321 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663325 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663329 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663334 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663338 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663343 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663347 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663353 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663358 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663362 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663368 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663372 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663376 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663381 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663385 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663390 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.676352 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663394 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663398 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663403 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663407 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663411 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663415 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663420 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663425 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663430 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663434 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663438 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663443 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663447 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663451 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663456 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663460 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663465 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663469 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663473 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663479 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.676897 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663486 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663491 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663495 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663500 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663505 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663509 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663513 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663526 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663531 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663535 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663539 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663542 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663546 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663550 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663555 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663559 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663562 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663567 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663571 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663575 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.677369 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663580 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663584 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663591 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663597 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663602 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663606 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663610 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663615 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663619 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663624 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663628 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663632 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663637 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663641 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663646 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663650 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663654 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663658 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663662 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.677853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663667 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663676 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663681 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663685 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663689 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663693 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663697 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663701 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663706 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663710 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663735 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663739 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663744 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663748 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663752 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663757 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663761 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.678347 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.663765 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.663808 5090 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664083 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664092 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664098 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664103 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664108 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664113 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664118 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664122 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664127 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664131 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664136 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664140 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664144 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.678825 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664148 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664153 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664158 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664162 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664167 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664171 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664175 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664179 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664184 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664188 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664192 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664198 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664205 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664210 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664215 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664219 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664223 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664227 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664231 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.679195 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664235 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664240 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664244 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664249 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664253 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664257 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664261 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664265 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664270 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664274 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664278 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664283 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664287 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664292 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664296 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664300 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664305 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664309 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664313 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664317 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.679656 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664321 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664328 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664333 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664338 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664342 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664346 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664351 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664355 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664359 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664364 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664368 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664372 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664377 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664381 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664386 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664390 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664394 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664401 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664405 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664409 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.680227 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664414 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664418 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664422 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664426 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664431 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664435 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664440 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664444 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664448 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664453 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664458 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664462 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664466 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664472 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664477 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664481 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664485 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664489 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664494 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.680687 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664499 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664503 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664510 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664515 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664520 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664525 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664530 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664534 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664538 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664543 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664549 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664553 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664558 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664562 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664566 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.664605 5090 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 01:19:59.681185 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664667 5090 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664673 5090 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664680 5090 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664685 5090 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664692 5090 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664697 5090 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664702 5090 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664707 5090 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664730 5090 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664735 5090 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664740 5090 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664744 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664749 5090 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664753 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664757 5090 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664761 5090 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664765 5090 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664770 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664774 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 01:19:59.681569 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664779 5090 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664783 5090 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664787 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664791 5090 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664795 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664799 5090 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664804 5090 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664817 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664822 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664826 5090 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664830 5090 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664834 5090 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664838 5090 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664842 5090 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664846 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664850 5090 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664854 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664859 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664863 5090 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 01:19:59.682055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664867 5090 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664874 5090 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664880 5090 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664885 5090 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664890 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664895 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664899 5090 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664903 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664907 5090 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664912 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664916 5090 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664920 5090 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664924 5090 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664928 5090 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664932 5090 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664936 5090 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664940 5090 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664945 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664949 5090 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664954 5090 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 01:19:59.682538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664958 5090 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664963 5090 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664967 5090 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664971 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664975 5090 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664979 5090 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664984 5090 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664988 5090 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664992 5090 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.664996 5090 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665000 5090 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665004 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665010 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665014 5090 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665019 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665031 5090 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665035 5090 feature_gate.go:386] unrecognized feature gate: Example Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665039 5090 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665043 5090 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665048 5090 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 01:19:59.683025 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665052 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665056 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665060 5090 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665064 5090 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665068 5090 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665073 5090 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665077 5090 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665081 5090 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665085 5090 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665090 5090 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665094 5090 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665099 5090 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665103 5090 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665107 5090 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665111 5090 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665116 5090 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665120 5090 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665125 5090 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665129 5090 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665133 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 01:19:59.683500 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665138 5090 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665143 5090 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665147 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665151 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665155 5090 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665159 5090 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665163 5090 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:19:59.665169 5090 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.665252 5090 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.666849 5090 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.666921 5090 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.667191 5090 server.go:1014] "Starting client certificate rotation" Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.667315 5090 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.667371 5090 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 01:14:18 +0000 UTC" deadline="2028-05-05 14:40:56.468001922 +0000 UTC" Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.667398 5090 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13261h20m56.800606078s" Oct 31 01:19:59.684008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.667919 5090 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 01:19:59.684362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.669755 5090 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 01:19:59.684362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.671282 5090 log.go:25] "Validated CRI v1 runtime API" Oct 31 01:19:59.684362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.672967 5090 log.go:25] "Validated CRI v1 image API" Oct 31 01:19:59.684362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.673775 5090 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 01:19:59.684362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.676540 5090 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cafca5fa-8b32-4e73-a54c-6a5b37301dd3:/dev/nvme0n1p3 cda684f1-2021-41db-b7f8-33cf0b76e508:/dev/nvme0n1p4] Oct 31 01:19:59.684520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.676559 5090 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/002abe69fdf0cff71a45630f8257a0c4691f81e8abd9f91943f7f008d427a6ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/002abe69fdf0cff71a45630f8257a0c4691f81e8abd9f91943f7f008d427a6ff/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/033b32812a0ee616cc654e355a148b6af9da2ab43bbc9361ba963d722fdbf468/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/033b32812a0ee616cc654e355a148b6af9da2ab43bbc9361ba963d722fdbf468/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0647a8d1532f7ed070bcac68f46da74acef76e2ac56ed35eec91aa94f9cdf2a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0647a8d1532f7ed070bcac68f46da74acef76e2ac56ed35eec91aa94f9cdf2a3/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10fd30ceae78ba7bb9f7a33c699cffea2273f788723cd2f013dc68a41ed9bba9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10fd30ceae78ba7bb9f7a33c699cffea2273f788723cd2f013dc68a41ed9bba9/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2dbb82f197cb32d0f5f2b32c07629da6c7cbbca22a8aec120f53b54b91ba192a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2dbb82f197cb32d0f5f2b32c07629da6c7cbbca22a8aec120f53b54b91ba192a/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a6c74706ec1df7d4df231c3bcae8579396870d64aac27547a1bb6b7f8a87986/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a6c74706ec1df7d4df231c3bcae8579396870d64aac27547a1bb6b7f8a87986/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49a684c2a440a507c27e2dbd30a18d4e943ee9f1165d9489437bf7e3a467b6b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49a684c2a440a507c27e2dbd30a18d4e943ee9f1165d9489437bf7e3a467b6b6/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/641250cc82e7a1a2a56baf6254d325710dc76f7c6ac3ac99f66e3dbc4a961382/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/641250cc82e7a1a2a56baf6254d325710dc76f7c6ac3ac99f66e3dbc4a961382/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d668c7e8a5b00abfdb515959e6d0a76472a63e6663a7e10bb03d88043fc23a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d668c7e8a5b00abfdb515959e6d0a76472a63e6663a7e10bb03d88043fc23a4/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/832394168dbe00bc7be0ea1b777b1528a2a81d660ac8fc4b55c020a9da8bed52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/832394168dbe00bc7be0ea1b777b1528a2a81d660ac8fc4b55c020a9da8bed52/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a91720bbb13b8af47c5132c2e04bb1d68c3696a9c1033f726569ad4a408db50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a91720bbb13b8af47c5132c2e04bb1d68c3696a9c1033f726569ad4a408db50/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/953f12dadbd965c75879b1c4ebaa5f5d57833f8ce929efdbc36699e769f68afa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/953f12dadbd965c75879b1c4ebaa5f5d57833f8ce929efdbc36699e769f68afa/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a1535e14a14cc97ff06a79a4502de33888575aeaa2baf6076a37e1b19f32181/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a1535e14a14cc97ff06a79a4502de33888575aeaa2baf6076a37e1b19f32181/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9bfad8dacd906348710ca485b77d8c72171891d94d3b397743397c2029eab74e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9bfad8dacd906348710ca485b77d8c72171891d94d3b397743397c2029eab74e/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a36700057e998a5b42a124389eb2d08769bf66d4a0943caa30a279671e469959/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a36700057e998a5b42a124389eb2d08769bf66d4a0943caa30a279671e469959/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5afaa159f5e08f6c39baed8b56a3bb5d973fecab7f63f2d0d87b39e60b8e627/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5afaa159f5e08f6c39baed8b56a3bb5d973fecab7f63f2d0d87b39e60b8e627/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5bfe3d8199224859cc7ff5a1cd6bfbab661a987ac0c45435dcaf395df1c53a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5bfe3d8199224859cc7ff5a1cd6bfbab661a987ac0c45435dcaf395df1c53a8/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1869ac005b5383a2cb2e06cffcb88f61c1bc8adb4b3dd6555dd8235288817df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1869ac005b5383a2cb2e06cffcb88f61c1bc8adb4b3dd6555dd8235288817df/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec19dd605f3028be1173c5ab62fc678af447d2a272e168ef8aa9d4fb0b7ae5c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec19dd605f3028be1173c5ab62fc678af447d2a272e168ef8aa9d4fb0b7ae5c5/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f31ee3b89016e3c5ebe8b3e83fda821ba97cad1b04fa8d0e9a36bd972333d0e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f31ee3b89016e3c5ebe8b3e83fda821ba97cad1b04fa8d0e9a36bd972333d0e4/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5c80534eec72c823422f44570f612e5d30ee4655cbbcabfcb61cb77451518bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5c80534eec72c823422f44570f612e5d30ee4655cbbcabfcb61cb77451518bc/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f69880c170d1166a954ac282d9b7cb1dade83cbf1a6ed9c6f6fb60b0857aa62f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f69880c170d1166a954ac282d9b7cb1dade83cbf1a6ed9c6f6fb60b0857aa62f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcc542ccc673cc228cb713f90fdb5af27c51aa0c14b88f4858356a429e5d0ddd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcc542ccc673cc228cb713f90fdb5af27c51aa0c14b88f4858356a429e5d0ddd/userdata/shm major:0 minor:315 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22fd2129-24b4-4c50-91be-bea78425e824/volumes/kubernetes.io~projected/kube-api-access-d2jhb:{mountpoint:/var/lib/kubelet/pods/22fd2129-24b4-4c50-91be-bea78425e824/volumes/kubernetes.io~projected/kube-api-access-d2jhb major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29151040-d2de-4a3c-b608-dc74c6aeee0e/volumes/kubernetes.io~projected/kube-api-access-rnjx7:{mountpoint:/var/lib/kubelet/pods/29151040-d2de-4a3c-b608-dc74c6aeee0e/volumes/kubernetes.io~projected/kube-api-access-rnjx7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e5f8170-98cf-4487-84e2-0aeccb45ba6c/volumes/kubernetes.io~projected/kube-api-access-bb74w:{mountpoint:/var/lib/kubelet/pods/3e5f8170-98cf-4487-84e2-0aeccb45ba6c/volumes/kubernetes.io~projected/kube-api-access-bb74w major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/43b11ef2-886e-4d57-9700-089f51fa360e/volumes/kubernetes.io~projected/kube-api-access-xphc2:{mountpoint:/var/lib/kubelet/pods/43b11ef2-886e-4d57-9700-089f51fa360e/volumes/kubernetes.io~projected/kube-api-access-xphc2 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4af660b2-97ac-4db7-b928-9d660b0c15a9/volumes/kubernetes.io~projected/kube-api-access-z66jj:{mountpoint:/var/lib/kubelet/pods/4af660b2-97ac-4db7-b928-9d660b0c15a9/volumes/kubernetes.io~projected/kube-api-access-z66jj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51211307-a4ca-4d06-95f7-02a7e5fef197/volumes/kubernetes.io~projected/kube-api-access-7tprw:{mountpoint:/var/lib/kubelet/pods/51211307-a4ca-4d06-95f7-02a7e5fef197/volumes/kubernetes.io~projected/kube-api-access-7tprw major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5472b622-19ab-4582-a5ef-4f09314f0213/volumes/kubernetes.io~projected/kube-api-access-rjvkm:{mountpoint:/var/lib/kubelet/pods/5472b622-19ab-4582-a5ef-4f09314f0213/volumes/kubernetes.io~projected/kube-api-access-rjvkm major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5472b622-19ab-4582-a5ef-4f09314f0213/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/5472b622-19ab-4582-a5ef-4f09314f0213/volumes/kubernetes.io~secret/signing-key major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/578616cb-95f4-49ef-afec-65f65cdb2f71/volumes/kubernetes.io~projected/kube-api-access-55prj:{mountpoint:/var/lib/kubelet/pods/578616cb-95f4-49ef-afec-65f65cdb2f71/volumes/kubernetes.io~projected/kube-api-access-55prj major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57ac8842-181e-4186-b7b4-5175a511a0e6/volumes/kubernetes.io~projected/kube-api-access-zclbj:{mountpoint:/var/lib/kubelet/pods/57ac8842-181e-4186-b7b4-5175a511a0e6/volumes/kubernetes.io~projected/kube-api-access-zclbj major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~empty-dir/tmp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~projected/kube-api-access-q5lxl:{mountpoint:/var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~projected/kube-api-access-q5lxl major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e6f6573-29b9-4b30-a82c-2b6e8948583f/volumes/kubernetes.io~projected/kube-api-access-ssr7p:{mountpoint:/var/lib/kubelet/pods/5e6f6573-29b9-4b30-a82c-2b6e8948583f/volumes/kubernetes.io~projected/kube-api-access-ssr7p major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6/volumes/kubernetes.io~projected/kube-api-access-rdtx7:{mountpoint:/var/lib/kubelet/pods/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6/volumes/kubernetes.io~projected/kube-api-access-rdtx7 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b85121a-782e-4f5a-b984-a775a806b231/volumes/kubernetes.io~projected/kube-api-access-jcmbh:{mountpoint:/var/lib/kubelet/pods/6b85121a-782e-4f5a-b984-a775a806b231/volumes/kubernetes.io~projected/kube-api-access-jcmbh major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volumes/kubernetes.io~projected/kube-api-access-lhkc6:{mountpoint:/var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volumes/kubernetes.io~projected/kube-api-access-lhkc6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e587446-856d-4580-a69f-94cd5953a81d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7e587446-856d-4580-a69f-94cd5953a81d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8280d35c-b526-41e0-b4a2-0bd74b181789/volumes/kubernetes.io~projected/kube-api-access-nb6xr:{mountpoint:/var/lib/kubelet/pods/8280d35c-b526-41e0-b4a2-0bd74b181789/volumes/kubernetes.io~projected/kube-api-access-nb6xr major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8280d35c-b526-41e0-b4a2-0bd74b181789/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8280d35c-b526-41e0-b4a2-0bd74b181789/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a3cb44a-89d0-4bb6-9815-3f52494b3047/volumes/kubernetes.io~projected/kube-api-access-gd88h:{mountpoint:/var/lib/kubelet/pods/8a3cb44a-89d0-4bb6-9815-3f52494b3047/volumes/kubernetes.io~projected/kube-api-access-gd88h major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a3cb44a-89d0-4bb6-9815-3f52494b3047/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8a3cb44a-89d0-4bb6-9815-3f52494b3047/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/930232e3-20a2-4134-99ab-9817dede3186/volumes/kubernetes.io~projected/kube-api-access-97bgp:{mountpoint:/var/lib/kubelet/pods/930232e3-20a2-4134-99ab-9817dede3186/volumes/kubernetes.io~projected/kube-api-access-97bgp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e634f3d-0490-480a-9484-a1ee415c5fb1/volumes/kubernetes.io~projected/kube-api-access-rvbbj:{mountpoint:/var/lib/kubelet/pods/9e634f3d-0490-480a-9484-a1ee415c5fb1/volumes/kubernetes.io~projected/kube-api-access-rvbbj major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fb8224e-3136-4261-8344-c4bc50dee44c/volumes/kubernetes.io~projected/kube-api-access-s5pxs:{mountpoint:/var/lib/kubelet/pods/9fb8224e-3136-4261-8344-c4bc50dee44c/volumes/kubernetes.io~projected/kube-api-access-s5pxs major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a489a94c-56c7-45df-92cb-42177dadd9c1/volumes/kubernetes.io~projected/kube-api-access-vjkg8:{mountpoint:/var/lib/kubelet/pods/a489a94c-56c7-45df-92cb-42177dadd9c1/volumes/kubernetes.io~projected/kube-api-access-vjkg8 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a489a94c-56c7-45df-92cb-42177dadd9c1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a489a94c-56c7-45df-92cb-42177dadd9c1/volumes/kubernetes.io~secret/cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~projected/kube-api-access-dxmkv:{mountpoint:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~projected/kube-api-access-dxmkv major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~projected/kube-api-access-cs9h6:{mountpoint:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~projected/kube-api-access-cs9h6 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~projected/kube-api-access-fp8hh:{mountpoint:/var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~projected/kube-api-access-fp8hh major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe/volumes/kubernetes.io~projected/kube-api-access-ndrdz:{mountpoint:/var/lib/kubelet/pods/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe/volumes/kubernetes.io~projected/kube-api-access-ndrdz major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd04b736-45e6-44cc-a032-e47ef93641a9/volumes/kubernetes.io~projected/kube-api-access-hkt44:{mountpoint:/var/lib/kubelet/pods/dd04b736-45e6-44cc-a032-e47ef93641a9/volumes/kubernetes.io~projected/kube-api-access-hkt44 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd04b736-45e6-44cc-a032-e47ef93641a9/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dd04b736-45e6-44cc-a032-e47ef93641a9/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df649d60-b1cd-408d-a0fc-e74fd86e6896/volumes/kubernetes.io~projected/kube-api-access-8shdj:{mountpoint:/var/lib/kubelet/pods/df649d60-b1cd-408d-a0fc-e74fd86e6896/volumes/kubernetes.io~projected/kube-api-access-8shdj major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9/volumes/kubernetes.io~projected/kube-api-access-4zdsp:{mountpoint:/var/lib/kubelet/pods/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9/volumes/kubernetes.io~projected/kube-api-access-4zdsp major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e137e152-115f-4459-80fe-a92048049da2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e137e152-115f-4459-80fe-a92048049da2/volumes/kubernetes.io~secret/agent-certs major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1b7733e-21c3-4770-bb58-d538802cbcaf/volumes/kubernetes.io~projected/kube-api-access-9bgq5:{mountpoint:/var/lib/kubelet/pods/f1b7733e-21c3-4770-bb58-d538802cbcaf/volumes/kubernetes.io~projected/kube-api-access-9bgq5 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f35a7382-ce08-44b1-8427-33a5d30252b7/volumes/kubernetes.io~projected/kube-api-access-rzgdm:{mountpoint:/var/lib/kubelet/pods/f35a7382-ce08-44b1-8427-33a5d30252b7/volumes/kubernetes.io~projected/kube-api-access-rzgdm major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f35a7382-ce08-44b1-8427-33a5d30252b7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f35a7382-ce08-44b1-8427-33a5d30252b7/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c/volumes/kubernetes.io~projected/kube-api-access-9dmsj:{mountpoint:/var/lib/kubelet/pods/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c/volumes/kubernetes.io~projected/kube-api-access-9dmsj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591/volumes/kubernetes.io~projected/kube-api-access-fdzdq:{mountpoint:/var/lib/kubelet/pods/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591/volumes/kubernetes.io~projected/kube-api-access-fdzdq major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/fd8b7e73920e5706e3f8412d8e21e85a0dd942640267dfc4ba5b20a9fb827635/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/19cf49c486ac001eddb5475060b7a3b8126abb6507e3a21a87db1cb19f19db6f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/923e9eb8f116fcbd7fe932b41c4bb35c4f9771f7f202c3a77b8887c17e4144a0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/28eb4e71bf38074871076a5a850bbc9fdc1f54f0a6b5b1f3a2b0a2521e7b00db/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/06780cf1ddada652fdffebe0d5c2cbee04477d335ea154a66a8312c11439dbad/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/a579735d9652e7c54d755f0d4d79081ab17b6fb8315462d8e090d2c754f43267/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f5fa2860aab93e323f848d62f114c29c765f05f6d1e84eab6e48634e022147ed/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3566f0d46de9126061fbcd4defcc4c3852499efad644a618069a2f1c9d623641/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/77fa026fa4869f3aa3660ccf14961b248350352f2c267f1725dde4cb6b54ee3a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0652d1eedb9bc7b9b90cbeb7839bf5f64f1afcfbdb248057746945cb4faf9dbf/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8fc9ce864919cda7de772c12a69b73f643ecad9353b81ed5e613c27c840afb30/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/4f1c882088e239341fc00b3dced713b64f37f3cdf02124480325fcadd90687e3/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/14cd0aeec5996a584484e161d0396d3130b259d4892f3d568f9d94fbea775d16/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5feed9dbdc1c0eedf32d22799a841a3be960e178786367fcac0122b55d3ae3f6/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/21e82942566a3cbc3de51a4847c5800e5f185071328917a82511b7d73b9c5502/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cfb510e00a978ce6f5400cce4724c434fe62997e9b3725beed50fb86204930d9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d006b13f723125fa9b6a57cbbafb887f19a33bbc9788a217de32069f508b1875/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/235e439a4661e10e0ad069c685720696db10060c4d73b9985548bb029d2ebad9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3fc535ce0ce2fa0287a2966804a195aeca226b5c7b69b04594821ebebf517b3f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d0e9e618ed0b1cae966fb07fe721be5246617943973147113cc7a7d4cbf1d957/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4dfa5879000838dbc45a5515b2e317c962863689aa11912d4c148e4780758da9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/72730402da3039596a99e2bdd7a18335a93b1656b8d3ebb0c53003225a0155d1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/438a652ab1c2d7701e224794facc127af8f955d2131037467822e806db171a9a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/c811e1c8034f3119b40b142b34e388d78ff5612c85aa75878a41f4ac0280729a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/bafe74e0c58935062e00dc9db07d7b3273b690d9cf8ff7b027fd6fda33f375a3/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f23e2db4ff2bad6e35c11f3bfacc41064c186491d4d0563136f2e084df3b4c60/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f7bd83ac7444d0ba37eb0621b673555c8b2d98750e23d95b110a5d66c0666670/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/0cd238f000d615438b4e99103d7bc34515d692a452ce980b72fcbaa4fdfe2491/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/19a5b6f598c685392ba71589fbf995d5cde85ae93747c40a0691433e903b4353/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/d6aa6d736b3d0d3ddaa716e3c629c6ad64ff8dc4e59918e1886bca97d64ab5ee/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/cf1b6ae0900805e6367d5b2c5995bad2a281d5ae624b0f9e0b8e43217a2ee855/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/0d57c284c7579fdfe5f7551bd696f7471eadef9500faf82663f955a133883845/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b15706726a185d903475b148610f87571bd3888e6637485aafc308ab372e7ab0/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/0cc39766a8e015ef187ee1db9f6b9b07cb18a2fff0d657948a3fb1dfd6dec682/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/75e867dd616bbafa0f7a4791e9efe1d17ba54fdb589adbd71232f244ec5a50e6/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/b55a8d5ba9e05dceb738af502416d0d4f16832ae281dd5678e0770ea94fad7ee/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/48b518668b56c3e17189ff42f110bca1f60d126fae92bc60aa624155bc2a023d/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/04d3f9c1a1335c198dc5c2fe80d41c582a1b1ff8895e2e483af95371b30d606a/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/ae9389207b30f4c2bfc0bec05a911a1005b26abba4f39930bf60a8aabd1443af/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/caf928adae6d14196016c1c08af34409008f14edbdad1d5566486234f7f10870/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/0d7c0eff30eb7740310d10023e1220bcd8676bee29cfca30d786a77f9fdb6e75/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/aa0ba203643ec6ac8bf9eed4c55d5f9d206f8776c78de4f39fdb99028ff2dc6b/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/ebbc1d52dc795f4cc98743ef52160fe3e9df1d40012ca007849eb764ff9e8d97/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/13cbf19075296785d8519a97d06154d66e63018af563ac13e9baf45b7741a89d/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/aee93024b424c42864fe36ae66ee51be09545ad9881a94e46b9570859c037f12/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/13ae2686c59cf805f88f437922d93372a14f1c602bd12e72f1dd30c05b385edd/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/5cf5b24c4927e7d7d677db0fc6f5d323267363926f8f12c556a08c617266e027/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/14380723d962a2b4812ab24269bf5ed8e8b33238a539fa64bef279318557d960/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/d8522717eb6fa5a9a22fb344caab377d044953a18a4b0687ba5935ef0cf8e2c1/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/c71d597533c9d45f7dda172a4156920ec5d2c1b015a5a750e86cd5421cec71a3/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/cccc7ec6cb731c4e9f4343e4e088a0f0be1221c77fcf9c50db0bdd3e1de612f5/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/f44b5e0d37b81ad5b6d1007c58729d290cfdf575d3376c89bd3eced525e59331/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/7788196b656fefc0b91c5447c3455c6de3d507c0920da7e1f664d445f95911fe/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/25c3fa672b02e73597c61392d95c2a466f7fca88f9c21c6ea971c59f7baf927c/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/3a3529cce75dadd9680e07841f3bc5804065c1c383881b1bcdd6ce308b78fd31/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/911b7b5e6114cf062989b4566a9e613817daee41ae325289f23383894ef4e525/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/05d123ebbccb6004150033245fc8d0a4af0ec2505152af9e9e0fbd91a3587e01/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4a37bbbfe3f2d6f4b310f86f116c80cc66d8e5464c67e8040621871d6998daad/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0b28c30c690b078115338f62f3aed756c85f35f94ec80f0cb7ba943b5887cc01/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4f87758b877d2bb648d679c66a4b25eb3defb0caa45e78865fde3437c76372dd/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/bb999b347d3745e97925f25ee329f7a1a03e573a95fb6936e7c0bae7239fa51a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/9bdc4694b1975b68df44b9b2e560e61914a4811e5c15587d58ac977e12b799fb/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 01:19:59.684925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.684558 5090 manager.go:217] Machine: {Timestamp:2026-10-31 01:19:59.6841009 +0000 UTC m=+0.064346114 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104836 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c567a0b7fa0c8e09ad7a0a141bb79 SystemUUID:ec2c567a-0b7f-a0c8-e09a-d7a0a141bb79 BootID:45835c78-5ce9-4a04-8024-a0d2f736b438 Filesystems:[{Device:/var/lib/kubelet/pods/8a3cb44a-89d0-4bb6-9815-3f52494b3047/volumes/kubernetes.io~projected/kube-api-access-gd88h DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd04b736-45e6-44cc-a032-e47ef93641a9/volumes/kubernetes.io~projected/kube-api-access-hkt44 DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10fd30ceae78ba7bb9f7a33c699cffea2273f788723cd2f013dc68a41ed9bba9/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/578616cb-95f4-49ef-afec-65f65cdb2f71/volumes/kubernetes.io~projected/kube-api-access-55prj DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d668c7e8a5b00abfdb515959e6d0a76472a63e6663a7e10bb03d88043fc23a4/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c/volumes/kubernetes.io~projected/kube-api-access-9dmsj DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f35a7382-ce08-44b1-8427-33a5d30252b7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f1b7733e-21c3-4770-bb58-d538802cbcaf/volumes/kubernetes.io~projected/kube-api-access-9bgq5 DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~projected/kube-api-access-cs9h6 DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a36700057e998a5b42a124389eb2d08769bf66d4a0943caa30a279671e469959/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9bfad8dacd906348710ca485b77d8c72171891d94d3b397743397c2029eab74e/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5afaa159f5e08f6c39baed8b56a3bb5d973fecab7f63f2d0d87b39e60b8e627/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/832394168dbe00bc7be0ea1b777b1528a2a81d660ac8fc4b55c020a9da8bed52/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51211307-a4ca-4d06-95f7-02a7e5fef197/volumes/kubernetes.io~projected/kube-api-access-7tprw DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a489a94c-56c7-45df-92cb-42177dadd9c1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2dbb82f197cb32d0f5f2b32c07629da6c7cbbca22a8aec120f53b54b91ba192a/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcc542ccc673cc228cb713f90fdb5af27c51aa0c14b88f4858356a429e5d0ddd/userdata/shm DeviceMajor:0 DeviceMinor:315 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49a684c2a440a507c27e2dbd30a18d4e943ee9f1165d9489437bf7e3a467b6b6/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a3cb44a-89d0-4bb6-9815-3f52494b3047/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volumes/kubernetes.io~projected/kube-api-access-lhkc6 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29151040-d2de-4a3c-b608-dc74c6aeee0e/volumes/kubernetes.io~projected/kube-api-access-rnjx7 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5472b622-19ab-4582-a5ef-4f09314f0213/volumes/kubernetes.io~projected/kube-api-access-rjvkm DeviceMajor:0 DeviceMinor:347 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e6f6573-29b9-4b30-a82c-2b6e8948583f/volumes/kubernetes.io~projected/kube-api-access-ssr7p DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/641250cc82e7a1a2a56baf6254d325710dc76f7c6ac3ac99f66e3dbc4a961382/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e587446-856d-4580-a69f-94cd5953a81d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe/volumes/kubernetes.io~projected/kube-api-access-ndrdz DeviceMajor:0 DeviceMinor:329 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591/volumes/kubernetes.io~projected/kube-api-access-fdzdq DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6/volumes/kubernetes.io~projected/kube-api-access-rdtx7 DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df649d60-b1cd-408d-a0fc-e74fd86e6896/volumes/kubernetes.io~projected/kube-api-access-8shdj DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b85121a-782e-4f5a-b984-a775a806b231/volumes/kubernetes.io~projected/kube-api-access-jcmbh DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a1535e14a14cc97ff06a79a4502de33888575aeaa2baf6076a37e1b19f32181/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~projected/kube-api-access-q5lxl DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5cf30068-f986-4286-b700-f68bdc0e5f75/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e137e152-115f-4459-80fe-a92048049da2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/43b11ef2-886e-4d57-9700-089f51fa360e/volumes/kubernetes.io~projected/kube-api-access-xphc2 DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4af660b2-97ac-4db7-b928-9d660b0c15a9/volumes/kubernetes.io~projected/kube-api-access-z66jj DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9fb8224e-3136-4261-8344-c4bc50dee44c/volumes/kubernetes.io~projected/kube-api-access-s5pxs DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8280d35c-b526-41e0-b4a2-0bd74b181789/volumes/kubernetes.io~projected/kube-api-access-nb6xr DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8280d35c-b526-41e0-b4a2-0bd74b181789/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f31ee3b89016e3c5ebe8b3e83fda821ba97cad1b04fa8d0e9a36bd972333d0e4/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec19dd605f3028be1173c5ab62fc678af447d2a272e168ef8aa9d4fb0b7ae5c5/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f35a7382-ce08-44b1-8427-33a5d30252b7/volumes/kubernetes.io~projected/kube-api-access-rzgdm DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~projected/kube-api-access-fp8hh DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/930232e3-20a2-4134-99ab-9817dede3186/volumes/kubernetes.io~projected/kube-api-access-97bgp DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/033b32812a0ee616cc654e355a148b6af9da2ab43bbc9361ba963d722fdbf468/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/57ac8842-181e-4186-b7b4-5175a511a0e6/volumes/kubernetes.io~projected/kube-api-access-zclbj DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f69880c170d1166a954ac282d9b7cb1dade83cbf1a6ed9c6f6fb60b0857aa62f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a91720bbb13b8af47c5132c2e04bb1d68c3696a9c1033f726569ad4a408db50/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9/volumes/kubernetes.io~projected/kube-api-access-4zdsp DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e5f8170-98cf-4487-84e2-0aeccb45ba6c/volumes/kubernetes.io~projected/kube-api-access-bb74w DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/002abe69fdf0cff71a45630f8257a0c4691f81e8abd9f91943f7f008d427a6ff/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5472b622-19ab-4582-a5ef-4f09314f0213/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd04b736-45e6-44cc-a032-e47ef93641a9/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22fd2129-24b4-4c50-91be-bea78425e824/volumes/kubernetes.io~projected/kube-api-access-d2jhb DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a489a94c-56c7-45df-92cb-42177dadd9c1/volumes/kubernetes.io~projected/kube-api-access-vjkg8 DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1869ac005b5383a2cb2e06cffcb88f61c1bc8adb4b3dd6555dd8235288817df/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9e634f3d-0490-480a-9484-a1ee415c5fb1/volumes/kubernetes.io~projected/kube-api-access-rvbbj DeviceMajor:0 DeviceMinor:308 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~projected/kube-api-access-dxmkv DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a6c74706ec1df7d4df231c3bcae8579396870d64aac27547a1bb6b7f8a87986/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5bfe3d8199224859cc7ff5a1cd6bfbab661a987ac0c45435dcaf395df1c53a8/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/befed7f0-b23a-48a5-803e-6f8bf380012c/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5c80534eec72c823422f44570f612e5d30ee4655cbbcabfcb61cb77451518bc/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0647a8d1532f7ed070bcac68f46da74acef76e2ac56ed35eec91aa94f9cdf2a3/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/953f12dadbd965c75879b1c4ebaa5f5d57833f8ce929efdbc36699e769f68afa/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/72564aed-4e32-4815-8a3e-6f61a8c36e83/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:002abe69fdf0cff MacAddress:72:84:3e:7c:29:d9 Speed:10000 Mtu:8901} {Name:0647a8d1532f7ed MacAddress:4a:3f:45:7c:5c:95 Speed:10000 Mtu:8901} {Name:10fd30ceae78ba7 MacAddress:d2:61:61:22:35:f8 Speed:10000 Mtu:8901} {Name:2dbb82f197cb32d MacAddress:b2:d6:17:fa:de:82 Speed:10000 Mtu:8901} {Name:6d668c7e8a5b00a MacAddress:c2:94:1f:0e:38:22 Speed:10000 Mtu:8901} {Name:832394168dbe00b MacAddress:62:84:94:f5:9e:e1 Speed:10000 Mtu:8901} {Name:9a1535e14a14cc9 MacAddress:0a:0e:f2:8f:cf:a2 Speed:10000 Mtu:8901} {Name:a36700057e998a5 MacAddress:4e:36:8d:29:6e:77 Speed:10000 Mtu:8901} {Name:b5afaa159f5e08f MacAddress:c2:0b:c8:af:af:d8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:86:28:f9 Speed:0 Mtu:9001} {Name:br-int MacAddress:56:6f:a8:2e:17:60 Speed:0 Mtu:8901} {Name:c1869ac005b5383 MacAddress:f2:82:0b:53:4a:6e Speed:10000 Mtu:8901} {Name:ec19dd605f3028b MacAddress:96:11:52:62:95:7e Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:86:28:f9 Speed:0 Mtu:9001} {Name:f31ee3b89016e3c MacAddress:5e:fc:f1:15:ab:30 Speed:10000 Mtu:8901} {Name:f4ba8108e6f24bc MacAddress:16:23:b3:ff:5e:30 Speed:10000 Mtu:8901} {Name:f5c80534eec72c8 MacAddress:02:24:7f:89:fe:60 Speed:10000 Mtu:8901} {Name:f69880c170d1166 MacAddress:52:36:c5:e8:8a:23 Speed:10000 Mtu:8901} {Name:fcc542ccc673cc2 MacAddress:c2:c5:e3:8c:60:62 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:26:98:81:06:4e:d2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:85:19:50:36:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 01:19:59.685112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.684930 5090 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 01:19:59.685112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.684975 5090 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 01:19:59.685296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685274 5090 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 01:19:59.685425 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685297 5090 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-145.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 01:19:59.685465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685427 5090 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 01:19:59.685465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685435 5090 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 01:19:59.685465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685445 5090 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 01:19:59.685465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685461 5090 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 01:19:59.685565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685480 5090 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 01:19:59.685643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685633 5090 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 01:19:59.685773 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685764 5090 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 01:19:59.685804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685801 5090 kubelet.go:493] "Attempting to sync node with API server" Oct 31 01:19:59.685830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685810 5090 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 01:19:59.685830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685821 5090 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 01:19:59.685830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685830 5090 kubelet.go:405] "Adding apiserver pod source" Oct 31 01:19:59.685921 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.685838 5090 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 01:19:59.686619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.686604 5090 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 01:19:59.686660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.686620 5090 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 01:19:59.687127 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.687108 5090 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 01:19:59.689435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689421 5090 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 01:19:59.689482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689438 5090 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 01:19:59.689552 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689541 5090 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 01:19:59.689808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689795 5090 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 01:19:59.689839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689827 5090 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 01:19:59.689942 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689932 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 01:19:59.689970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689949 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 01:19:59.689970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689955 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 01:19:59.690042 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.689963 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 01:19:59.690042 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690024 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 01:19:59.690042 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690033 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 01:19:59.690042 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690041 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 01:19:59.690155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690050 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 01:19:59.690155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690058 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 01:19:59.690155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690064 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 01:19:59.690155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690073 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 01:19:59.690155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690082 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 01:19:59.690155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690108 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 01:19:59.690155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.690117 5090 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 01:19:59.693684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.693672 5090 server.go:1263] "Started kubelet" Oct 31 01:19:59.693778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.693749 5090 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 01:19:59.694426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.693845 5090 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 01:19:59.694426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.694242 5090 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 01:19:59.694530 ip-10-0-141-145 systemd[1]: Started Kubernetes Kubelet. Oct 31 01:19:59.699754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.696039 5090 server.go:317] "Adding debug handlers to kubelet server" Oct 31 01:19:59.699754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.696605 5090 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 01:19:59.699929 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.699832 5090 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 01:19:59.700150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.700090 5090 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 01:19:59.700150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.700100 5090 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 01:19:59.700317 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:19:59.700295 5090 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Oct 31 01:19:59.700625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.700595 5090 factory.go:55] Registering systemd factory Oct 31 01:19:59.700625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.700615 5090 factory.go:223] Registration of the systemd container factory successfully Oct 31 01:19:59.700806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.700090 5090 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 01:19:59.700806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.700792 5090 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 01:19:59.701296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.701271 5090 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 01:14:19 +0000 UTC" deadline="2028-04-15 02:19:22.438782677 +0000 UTC" Oct 31 01:19:59.701380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.701298 5090 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12768h59m22.737488622s" Oct 31 01:19:59.702962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.701925 5090 factory.go:153] Registering CRI-O factory Oct 31 01:19:59.702962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.701946 5090 factory.go:223] Registration of the crio container factory successfully Oct 31 01:19:59.702962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.702020 5090 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 01:19:59.702962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.702039 5090 factory.go:103] Registering Raw factory Oct 31 01:19:59.702962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.702053 5090 manager.go:1199] Started watching for new ooms in manager Oct 31 01:19:59.702962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.702412 5090 manager.go:319] Starting recovery of all containers Oct 31 01:19:59.704137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704106 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="930232e3-20a2-4134-99ab-9817dede3186" volumeName="kubernetes.io/empty-dir/930232e3-20a2-4134-99ab-9817dede3186-tmp-dir" seLinuxMountContext="" Oct 31 01:19:59.704173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704142 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb8224e-3136-4261-8344-c4bc50dee44c" volumeName="kubernetes.io/configmap/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-config" seLinuxMountContext="" Oct 31 01:19:59.704173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704155 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b123e4c4-57cd-49f8-985b-158b4deab82d" volumeName="kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration" seLinuxMountContext="" Oct 31 01:19:59.704173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704168 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af660b2-97ac-4db7-b928-9d660b0c15a9" volumeName="kubernetes.io/projected/4af660b2-97ac-4db7-b928-9d660b0c15a9-kube-api-access-z66jj" seLinuxMountContext="" Oct 31 01:19:59.704267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704181 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b85121a-782e-4f5a-b984-a775a806b231" volumeName="kubernetes.io/projected/6b85121a-782e-4f5a-b984-a775a806b231-kube-api-access-jcmbh" seLinuxMountContext="" Oct 31 01:19:59.704267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704193 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd51c16f-f327-4939-ad47-4acc6aaf62c2" volumeName="kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6" seLinuxMountContext="" Oct 31 01:19:59.704267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704205 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd51c16f-f327-4939-ad47-4acc6aaf62c2" volumeName="kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets" seLinuxMountContext="" Oct 31 01:19:59.704267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704216 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35a7382-ce08-44b1-8427-33a5d30252b7" volumeName="kubernetes.io/secret/f35a7382-ce08-44b1-8427-33a5d30252b7-serving-cert" seLinuxMountContext="" Oct 31 01:19:59.704267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704230 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf30068-f986-4286-b700-f68bdc0e5f75" volumeName="kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-tuned" seLinuxMountContext="" Oct 31 01:19:59.704267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704259 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c" volumeName="kubernetes.io/projected/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kube-api-access-9dmsj" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704272 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e5f8170-98cf-4487-84e2-0aeccb45ba6c" volumeName="kubernetes.io/projected/3e5f8170-98cf-4487-84e2-0aeccb45ba6c-kube-api-access-bb74w" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704287 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5472b622-19ab-4582-a5ef-4f09314f0213" volumeName="kubernetes.io/projected/5472b622-19ab-4582-a5ef-4f09314f0213-kube-api-access-rjvkm" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704303 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd51c16f-f327-4939-ad47-4acc6aaf62c2" volumeName="kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704319 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1b7733e-21c3-4770-bb58-d538802cbcaf" volumeName="kubernetes.io/projected/f1b7733e-21c3-4770-bb58-d538802cbcaf-kube-api-access-9bgq5" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704346 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29151040-d2de-4a3c-b608-dc74c6aeee0e" volumeName="kubernetes.io/projected/29151040-d2de-4a3c-b608-dc74c6aeee0e-kube-api-access-rnjx7" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704358 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf30068-f986-4286-b700-f68bdc0e5f75" volumeName="kubernetes.io/projected/5cf30068-f986-4286-b700-f68bdc0e5f75-kube-api-access-q5lxl" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704370 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e634f3d-0490-480a-9484-a1ee415c5fb1" volumeName="kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704381 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e634f3d-0490-480a-9484-a1ee415c5fb1" volumeName="kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704393 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a489a94c-56c7-45df-92cb-42177dadd9c1" volumeName="kubernetes.io/secret/a489a94c-56c7-45df-92cb-42177dadd9c1-cert" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704404 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b123e4c4-57cd-49f8-985b-158b4deab82d" volumeName="kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704416 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd51c16f-f327-4939-ad47-4acc6aaf62c2" volumeName="kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704428 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befed7f0-b23a-48a5-803e-6f8bf380012c" volumeName="kubernetes.io/projected/befed7f0-b23a-48a5-803e-6f8bf380012c-kube-api-access-fp8hh" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704441 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd51c16f-f327-4939-ad47-4acc6aaf62c2" volumeName="kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704452 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bf687ff-9445-49b0-804a-5eb5ca7ddedc" volumeName="kubernetes.io/configmap/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-nginx-conf" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704465 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af660b2-97ac-4db7-b928-9d660b0c15a9" volumeName="kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-binary-copy" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704476 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8280d35c-b526-41e0-b4a2-0bd74b181789" volumeName="kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-tmp" seLinuxMountContext="" Oct 31 01:19:59.704485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704488 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befed7f0-b23a-48a5-803e-6f8bf380012c" volumeName="kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-stats-auth" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704502 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd51cb1-85bf-472e-b0d7-14b6bf1cf591" volumeName="kubernetes.io/configmap/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-iptables-alerter-script" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704545 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29151040-d2de-4a3c-b608-dc74c6aeee0e" volumeName="kubernetes.io/configmap/29151040-d2de-4a3c-b608-dc74c6aeee0e-serviceca" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704616 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="43b11ef2-886e-4d57-9700-089f51fa360e" volumeName="kubernetes.io/projected/43b11ef2-886e-4d57-9700-089f51fa360e-kube-api-access-xphc2" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704755 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5472b622-19ab-4582-a5ef-4f09314f0213" volumeName="kubernetes.io/configmap/5472b622-19ab-4582-a5ef-4f09314f0213-signing-cabundle" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704769 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8280d35c-b526-41e0-b4a2-0bd74b181789" volumeName="kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-trusted-ca-bundle" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704780 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b123e4c4-57cd-49f8-985b-158b4deab82d" volumeName="kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704792 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b123e4c4-57cd-49f8-985b-158b4deab82d" volumeName="kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704806 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51211307-a4ca-4d06-95f7-02a7e5fef197" volumeName="kubernetes.io/projected/51211307-a4ca-4d06-95f7-02a7e5fef197-kube-api-access-7tprw" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704818 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="578616cb-95f4-49ef-afec-65f65cdb2f71" volumeName="kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704829 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8280d35c-b526-41e0-b4a2-0bd74b181789" volumeName="kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-snapshots" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704841 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e634f3d-0490-480a-9484-a1ee415c5fb1" volumeName="kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704854 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a489a94c-56c7-45df-92cb-42177dadd9c1" volumeName="kubernetes.io/projected/a489a94c-56c7-45df-92cb-42177dadd9c1-kube-api-access-vjkg8" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704866 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b123e4c4-57cd-49f8-985b-158b4deab82d" volumeName="kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704881 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd51c16f-f327-4939-ad47-4acc6aaf62c2" volumeName="kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704895 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35a7382-ce08-44b1-8427-33a5d30252b7" volumeName="kubernetes.io/configmap/f35a7382-ce08-44b1-8427-33a5d30252b7-config" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704906 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af660b2-97ac-4db7-b928-9d660b0c15a9" volumeName="kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704920 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8280d35c-b526-41e0-b4a2-0bd74b181789" volumeName="kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-service-ca-bundle" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704933 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b123e4c4-57cd-49f8-985b-158b4deab82d" volumeName="kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704945 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5472b622-19ab-4582-a5ef-4f09314f0213" volumeName="kubernetes.io/secret/5472b622-19ab-4582-a5ef-4f09314f0213-signing-key" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704960 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd04b736-45e6-44cc-a032-e47ef93641a9" volumeName="kubernetes.io/secret/dd04b736-45e6-44cc-a032-e47ef93641a9-serving-cert" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704974 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df649d60-b1cd-408d-a0fc-e74fd86e6896" volumeName="kubernetes.io/empty-dir/df649d60-b1cd-408d-a0fc-e74fd86e6896-tmp-dir" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704987 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e137e152-115f-4459-80fe-a92048049da2" volumeName="kubernetes.io/configmap/e137e152-115f-4459-80fe-a92048049da2-konnectivity-ca" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.704999 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f35a7382-ce08-44b1-8427-33a5d30252b7" volumeName="kubernetes.io/projected/f35a7382-ce08-44b1-8427-33a5d30252b7-kube-api-access-rzgdm" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705013 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b85121a-782e-4f5a-b984-a775a806b231" volumeName="kubernetes.io/configmap/6b85121a-782e-4f5a-b984-a775a806b231-telemetry-config" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705026 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e587446-856d-4580-a69f-94cd5953a81d" volumeName="kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705039 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="930232e3-20a2-4134-99ab-9817dede3186" volumeName="kubernetes.io/projected/930232e3-20a2-4134-99ab-9817dede3186-kube-api-access-97bgp" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705051 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd51c16f-f327-4939-ad47-4acc6aaf62c2" volumeName="kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705064 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="befed7f0-b23a-48a5-803e-6f8bf380012c" volumeName="kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-default-certificate" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705078 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfa8037e-1fcd-4d2f-b8d5-edc99be74be9" volumeName="kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-daemon-config" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705138 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5cf30068-f986-4286-b700-f68bdc0e5f75" volumeName="kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-tmp" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705154 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3cb44a-89d0-4bb6-9815-3f52494b3047" volumeName="kubernetes.io/configmap/8a3cb44a-89d0-4bb6-9815-3f52494b3047-config" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705167 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e634f3d-0490-480a-9484-a1ee415c5fb1" volumeName="kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705180 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df649d60-b1cd-408d-a0fc-e74fd86e6896" volumeName="kubernetes.io/configmap/df649d60-b1cd-408d-a0fc-e74fd86e6896-config-volume" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705191 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df649d60-b1cd-408d-a0fc-e74fd86e6896" volumeName="kubernetes.io/projected/df649d60-b1cd-408d-a0fc-e74fd86e6896-kube-api-access-8shdj" seLinuxMountContext="" Oct 31 01:19:59.705225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705205 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8280d35c-b526-41e0-b4a2-0bd74b181789" volumeName="kubernetes.io/projected/8280d35c-b526-41e0-b4a2-0bd74b181789-kube-api-access-nb6xr" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705216 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3cb44a-89d0-4bb6-9815-3f52494b3047" volumeName="kubernetes.io/projected/8a3cb44a-89d0-4bb6-9815-3f52494b3047-kube-api-access-gd88h" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705503 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a3cb44a-89d0-4bb6-9815-3f52494b3047" volumeName="kubernetes.io/secret/8a3cb44a-89d0-4bb6-9815-3f52494b3047-serving-cert" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705524 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fb8224e-3136-4261-8344-c4bc50dee44c" volumeName="kubernetes.io/projected/9fb8224e-3136-4261-8344-c4bc50dee44c-kube-api-access-s5pxs" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705544 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b123e4c4-57cd-49f8-985b-158b4deab82d" volumeName="kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705556 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9da61d-3af1-4aa1-8ddd-de96b78ffafe" volumeName="kubernetes.io/empty-dir/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-data-volume" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705570 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd04b736-45e6-44cc-a032-e47ef93641a9" volumeName="kubernetes.io/projected/dd04b736-45e6-44cc-a032-e47ef93641a9-kube-api-access-hkt44" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705586 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfa8037e-1fcd-4d2f-b8d5-edc99be74be9" volumeName="kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cni-binary-copy" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705600 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ae589ef-bc72-4b28-a05a-14dc01aa1dd6" volumeName="kubernetes.io/projected/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6-kube-api-access-rdtx7" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705621 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72564aed-4e32-4815-8a3e-6f61a8c36e83" volumeName="kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-env-overrides" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705636 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd04b736-45e6-44cc-a032-e47ef93641a9" volumeName="kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-config" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705655 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfa8037e-1fcd-4d2f-b8d5-edc99be74be9" volumeName="kubernetes.io/projected/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-kube-api-access-4zdsp" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705669 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e137e152-115f-4459-80fe-a92048049da2" volumeName="kubernetes.io/secret/e137e152-115f-4459-80fe-a92048049da2-agent-certs" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705685 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af660b2-97ac-4db7-b928-9d660b0c15a9" volumeName="kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705708 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72564aed-4e32-4815-8a3e-6f61a8c36e83" volumeName="kubernetes.io/projected/72564aed-4e32-4815-8a3e-6f61a8c36e83-kube-api-access-lhkc6" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705750 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72564aed-4e32-4815-8a3e-6f61a8c36e83" volumeName="kubernetes.io/secret/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705769 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8280d35c-b526-41e0-b4a2-0bd74b181789" volumeName="kubernetes.io/secret/8280d35c-b526-41e0-b4a2-0bd74b181789-serving-cert" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705782 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9da61d-3af1-4aa1-8ddd-de96b78ffafe" volumeName="kubernetes.io/projected/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-api-access-ndrdz" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705792 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffd51cb1-85bf-472e-b0d7-14b6bf1cf591" volumeName="kubernetes.io/projected/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-kube-api-access-fdzdq" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705804 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22fd2129-24b4-4c50-91be-bea78425e824" volumeName="kubernetes.io/projected/22fd2129-24b4-4c50-91be-bea78425e824-kube-api-access-d2jhb" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705813 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57ac8842-181e-4186-b7b4-5175a511a0e6" volumeName="kubernetes.io/projected/57ac8842-181e-4186-b7b4-5175a511a0e6-kube-api-access-zclbj" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705825 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e6f6573-29b9-4b30-a82c-2b6e8948583f" volumeName="kubernetes.io/projected/5e6f6573-29b9-4b30-a82c-2b6e8948583f-kube-api-access-ssr7p" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705834 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72564aed-4e32-4815-8a3e-6f61a8c36e83" volumeName="kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-script-lib" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705843 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72564aed-4e32-4815-8a3e-6f61a8c36e83" volumeName="kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-config" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705855 5090 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd04b736-45e6-44cc-a032-e47ef93641a9" volumeName="kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-trusted-ca" seLinuxMountContext="" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705864 5090 reconstruct.go:97] "Volume reconstruction finished" Oct 31 01:19:59.706679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.705869 5090 reconciler.go:29] "Reconciler: start to sync state" Oct 31 01:19:59.710437 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.710418 5090 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:59.711248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.711007 5090 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:59.714570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.713056 5090 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:59.715518 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.715481 5090 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 01:19:59.730486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.730309 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:19:59.734177 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.734153 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:19:59.736782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.736693 5090 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 01:19:59.737709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.737692 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:19:59.738410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.738392 5090 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 01:19:59.738492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.738419 5090 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 01:19:59.738492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.738440 5090 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 01:19:59.738492 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:19:59.738481 5090 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 01:19:59.741494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.741457 5090 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 01:19:59.742695 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.742678 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:19:59.754335 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.754316 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:19:59.754963 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.754913 5090 generic.go:356] "Generic (PLEG): container finished" podID="72564aed-4e32-4815-8a3e-6f61a8c36e83" containerID="2c9576f4d1933fdfa68e83b30f3f03d125afc002f39d3ce0abc79cbbdb5bbc84" exitCode=1 Oct 31 01:19:59.760317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.760292 5090 generic.go:356] "Generic (PLEG): container finished" podID="92683d17f4ef39773bb8bc1b1459025a" containerID="4ab65acd0d1debd956d57d811c58177a29b03e43a22a1d924a5141436afadcbd" exitCode=0 Oct 31 01:19:59.765738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.765693 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:19:59.765843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.765770 5090 generic.go:356] "Generic (PLEG): container finished" podID="dd04b736-45e6-44cc-a032-e47ef93641a9" containerID="176f3b933e5201fea082acd69c7789423a6693b3fe23407901e56699c4c6cc14" exitCode=255 Oct 31 01:19:59.791821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.791748 5090 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="8c93f3d9d0b3c7bc7602b70aab4e95762fee5498cf8e6e805cd976a1f007e82f" exitCode=0 Oct 31 01:19:59.791821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.791774 5090 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="09a9e2f642d305e0fd11f1c7027e2578fe201c2c57c6d7eab84d0cfed1b3d4f3" exitCode=0 Oct 31 01:19:59.791821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.791782 5090 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="3f50a2fbcb84affe7d79e03b51001caa779d29d4f42edc1dc4c1b8e692fa11f1" exitCode=0 Oct 31 01:19:59.791821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.791789 5090 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="e8281205d302897656d8ce6576ecfa64cacfaa09396306c91d139262e9ac6dcd" exitCode=0 Oct 31 01:19:59.791821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.791796 5090 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="6e3dfee6702f03a54075282d331802fc604a09cd6e70ce4a1cc3d76501343b37" exitCode=0 Oct 31 01:19:59.791821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.791802 5090 generic.go:356] "Generic (PLEG): container finished" podID="4af660b2-97ac-4db7-b928-9d660b0c15a9" containerID="3b2fd26ef884fcb409e8493f6b1b9fbd75bab8fb713c8a98e42f91275b38fee0" exitCode=0 Oct 31 01:19:59.811117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.811093 5090 manager.go:324] Recovery completed Oct 31 01:19:59.831117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831094 5090 cpu_manager.go:226] "Starting" policy="none" Oct 31 01:19:59.831117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831112 5090 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 01:19:59.831117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831130 5090 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 01:19:59.831289 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831274 5090 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 01:19:59.831334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831287 5090 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 01:19:59.831334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831315 5090 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 01:19:59.831334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831321 5090 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 01:19:59.831334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831326 5090 policy_none.go:50] "Start" Oct 31 01:19:59.831441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831340 5090 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 01:19:59.831441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831349 5090 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 01:19:59.831492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831486 5090 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 01:19:59.831525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831494 5090 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 01:19:59.831525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.831500 5090 policy_none.go:44] "Start" Oct 31 01:19:59.836192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.836161 5090 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 01:19:59.838879 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:19:59.838857 5090 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 01:19:59.838998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.838958 5090 manager.go:341] "Starting Device Plugin manager" Oct 31 01:19:59.839572 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:19:59.839556 5090 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 01:19:59.839684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.839574 5090 server.go:87] "Starting device plugin registration server" Oct 31 01:19:59.839839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.839827 5090 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 01:19:59.839878 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.839841 5090 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 01:19:59.839985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.839972 5090 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 01:19:59.840284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.840171 5090 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 01:19:59.840400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.840294 5090 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 01:19:59.840400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.840302 5090 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 01:19:59.840553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.840380 5090 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T01:19:59.840281336Z","UUID":"6e15a35c-57d8-41b8-9a0c-e8e1c1dc061f","Handler":null,"Name":"","Endpoint":""} Oct 31 01:19:59.842274 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:19:59.842247 5090 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 01:19:59.843752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.843733 5090 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 01:19:59.843893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.843883 5090 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 01:19:59.940771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.940741 5090 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 01:19:59.942022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.942005 5090 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 01:19:59.942108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.942034 5090 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 01:19:59.942108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.942048 5090 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 01:19:59.942108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.942099 5090 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:59.955516 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.955496 5090 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:19:59.955615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:19:59.955552 5090 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.039444 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.039382 5090 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal"] Oct 31 01:20:00.040035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.039998 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"a09dffb959e1d2fc35bb443f526373c0b67a163f599523498344b922dd35e20e"} Oct 31 01:20:00.040082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.040045 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"34ad3890669a849498fe59e15bc7e794fc0904f61ee9df3cf4ec56303b357344"} Oct 31 01:20:00.040082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.040059 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"2652a7b235c1e80a3bbd8e448e296f345a76e8b40a75349a743da54ebfb227f1"} Oct 31 01:20:00.040082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.040061 5090 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.040082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.040073 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerDied","Data":"4ab65acd0d1debd956d57d811c58177a29b03e43a22a1d924a5141436afadcbd"} Oct 31 01:20:00.040210 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.040092 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"cdb91b62e8cf7b4706597a8ccfd5b5e604af5e586c51ea1d2a6c7b89fb53bbaf"} Oct 31 01:20:00.040254 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.040211 5090 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.053917 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.053867 5090 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 01:20:00.053917 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.053912 5090 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.054575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.054557 5090 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 01:20:00.054631 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.054596 5090 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.117723 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.117696 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.117818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.117743 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.117849 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.117817 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.218345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.218268 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.218473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.218362 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.218473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.218379 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.218473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.218323 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.218473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.218425 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.218673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.218473 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.686971 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.686933 5090 apiserver.go:52] "Watching apiserver" Oct 31 01:20:00.695861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.695831 5090 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 01:20:00.698830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.697272 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-kjrt4","kube-system/konnectivity-agent-7rq4g","openshift-image-registry/image-registry-fb9bcc569-v7h6k","openshift-multus/network-metrics-daemon-wff52","openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2","openshift-network-operator/iptables-alerter-fcqc8","openshift-image-registry/node-ca-pmc4f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal","openshift-ingress/router-default-54786d95-fr8g9","openshift-ingress-canary/ingress-canary-jn8kf","openshift-multus/multus-xfqs2","kube-system/global-pull-secret-syncer-c5d98","openshift-image-registry/image-registry-54d575cd44-5jzvn","kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal","openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf","openshift-network-diagnostics/network-check-target-klvct","kube-system/kas-connection-checker-685895c7c9-d9zxv","openshift-insights/insights-runtime-extractor-8jv29","openshift-network-console/networking-console-plugin-86c478bb88-d74bn","kube-system/kas-connection-checker-685895c7c9-llbl2","openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg","openshift-cluster-node-tuning-operator/tuned-bf87z","kube-system/kas-connection-checker-685895c7c9-bcgq4","openshift-dns/dns-default-mrqzl","openshift-multus/multus-additional-cni-plugins-h7k55","openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps","openshift-insights/insights-operator-7c4986d449-7hl7l","openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7","openshift-service-ca/service-ca-5fffc97bdf-bw6g8","openshift-dns/node-resolver-mz75q","openshift-console-operator/console-operator-7cd56b8dbc-zkw9h","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452","openshift-insights/periodic-gathering-slnps-lhfzm"] Oct 31 01:20:00.700533 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.700510 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:00.702178 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.702159 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.702313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.702292 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.702381 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.702318 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.703305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.703287 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:00.703799 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.703780 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.704340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.704317 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.704340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.704335 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.704495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.704416 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:00.704894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.704878 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 01:20:00.704966 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.704946 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.705152 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.705137 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.705614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.705600 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 01:20:00.705695 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.705643 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 01:20:00.706000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.705981 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:20:00.708281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.708258 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.708455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.708437 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 01:20:00.708515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.708468 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 01:20:00.708515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.708476 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 01:20:00.708515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.708511 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9clxs\"" Oct 31 01:20:00.708802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.708784 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qvjmz\"" Oct 31 01:20:00.709008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.708980 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 01:20:00.709110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709008 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gr6nh\"" Oct 31 01:20:00.709110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709040 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 01:20:00.709110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709061 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 01:20:00.709110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709084 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.709333 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709225 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.709333 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709265 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 01:20:00.709425 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709397 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ctd5c\"" Oct 31 01:20:00.709590 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709565 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 01:20:00.709749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.709732 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-qjxmz\"" Oct 31 01:20:00.713272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.713248 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 01:20:00.714328 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.714306 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 01:20:00.714483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.714468 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.714601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.714578 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 01:20:00.714703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.714689 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 01:20:00.714872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.714851 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.714927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.714896 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 01:20:00.715078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.715061 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.715160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.714900 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.715257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.715220 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 01:20:00.715498 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.715479 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.715807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.715787 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 01:20:00.715807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.715803 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.715934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.715830 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 01:20:00.716027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.716008 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.716196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.716177 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.716246 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.716213 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 01:20:00.716312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.716296 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xp7zg\"" Oct 31 01:20:00.716473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.716455 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.716551 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.716531 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.716811 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.716795 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.717074 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717055 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.717140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717089 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 01:20:00.717368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717352 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lbv59\"" Oct 31 01:20:00.717575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717556 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 01:20:00.717650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717632 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.717802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717785 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-q7mlm\"" Oct 31 01:20:00.717869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717841 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.717869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.717856 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.718080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.718062 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 01:20:00.718488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.718465 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.718562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.718505 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vj55m\"" Oct 31 01:20:00.718562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.718529 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.719360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719278 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 01:20:00.719360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719318 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.719360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719328 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.719527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719372 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 01:20:00.719527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719393 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 01:20:00.719527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719424 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fnh8f\"" Oct 31 01:20:00.719527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719469 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.719527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719497 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.719858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719553 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 01:20:00.719858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719558 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.719858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719604 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vjbgx\"" Oct 31 01:20:00.719858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719664 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 01:20:00.719858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719766 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.719858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719843 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lqw4r\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719867 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719923 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719936 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.719936 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gv7wl\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720010 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z69qs\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720046 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720071 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hm2lp\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720100 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-p75wq\"" Oct 31 01:20:00.720120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720123 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720125 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tsbmr\"" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720148 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgq5\" (UniqueName: \"kubernetes.io/projected/f1b7733e-21c3-4770-bb58-d538802cbcaf-kube-api-access-9bgq5\") pod \"kas-connection-checker-685895c7c9-d9zxv\" (UID: \"f1b7733e-21c3-4770-bb58-d538802cbcaf\") " pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720182 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclbj\" (UniqueName: \"kubernetes.io/projected/57ac8842-181e-4186-b7b4-5175a511a0e6-kube-api-access-zclbj\") pod \"kas-connection-checker-685895c7c9-bcgq4\" (UID: \"57ac8842-181e-4186-b7b4-5175a511a0e6\") " pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720211 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2jhb\" (UniqueName: \"kubernetes.io/projected/22fd2129-24b4-4c50-91be-bea78425e824-kube-api-access-d2jhb\") pod \"kas-connection-checker-685895c7c9-llbl2\" (UID: \"22fd2129-24b4-4c50-91be-bea78425e824\") " pod="kube-system/kas-connection-checker-685895c7c9-llbl2" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720241 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h7598\"" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720306 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wq6mh\"" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720310 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720240 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b85121a-782e-4f5a-b984-a775a806b231-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720358 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720390 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmbh\" (UniqueName: \"kubernetes.io/projected/6b85121a-782e-4f5a-b984-a775a806b231-kube-api-access-jcmbh\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:00.720548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720505 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b85121a-782e-4f5a-b984-a775a806b231-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:00.721071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720562 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cv59q\"" Oct 31 01:20:00.721071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720659 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 01:20:00.721071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720699 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.721071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720742 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.721071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720810 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 01:20:00.721071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720871 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 01:20:00.721071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.720910 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82mbc\"" Oct 31 01:20:00.721845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.721664 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 01:20:00.721845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.721672 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.721845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.721698 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.721845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.721801 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 01:20:00.722083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.721950 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5hw4\"" Oct 31 01:20:00.722083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.721990 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j25cw\"" Oct 31 01:20:00.722653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.722635 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 01:20:00.723402 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.723383 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-whdn9\"" Oct 31 01:20:00.723512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.723416 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 01:20:00.723512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.723433 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 01:20:00.723724 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.723701 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-79nk6\"" Oct 31 01:20:00.723794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.723737 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 01:20:00.724984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.724945 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bztmp\"" Oct 31 01:20:00.725198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725125 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.725198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725133 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.725198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725155 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.725198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725180 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.725198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725189 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4ls7t\"" Oct 31 01:20:00.725463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725206 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 01:20:00.725463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725267 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 01:20:00.725463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725281 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 01:20:00.725463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725434 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dnq6d\"" Oct 31 01:20:00.725656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725563 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 01:20:00.725656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725584 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 01:20:00.725656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.725620 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 01:20:00.730278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.730258 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgq5\" (UniqueName: \"kubernetes.io/projected/f1b7733e-21c3-4770-bb58-d538802cbcaf-kube-api-access-9bgq5\") pod \"kas-connection-checker-685895c7c9-d9zxv\" (UID: \"f1b7733e-21c3-4770-bb58-d538802cbcaf\") " pod="kube-system/kas-connection-checker-685895c7c9-d9zxv" Oct 31 01:20:00.730373 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.730357 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2jhb\" (UniqueName: \"kubernetes.io/projected/22fd2129-24b4-4c50-91be-bea78425e824-kube-api-access-d2jhb\") pod \"kas-connection-checker-685895c7c9-llbl2\" (UID: \"22fd2129-24b4-4c50-91be-bea78425e824\") " pod="kube-system/kas-connection-checker-685895c7c9-llbl2" Oct 31 01:20:00.730811 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.730788 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclbj\" (UniqueName: \"kubernetes.io/projected/57ac8842-181e-4186-b7b4-5175a511a0e6-kube-api-access-zclbj\") pod \"kas-connection-checker-685895c7c9-bcgq4\" (UID: \"57ac8842-181e-4186-b7b4-5175a511a0e6\") " pod="kube-system/kas-connection-checker-685895c7c9-bcgq4" Oct 31 01:20:00.731095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.731080 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmbh\" (UniqueName: \"kubernetes.io/projected/6b85121a-782e-4f5a-b984-a775a806b231-kube-api-access-jcmbh\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:00.734867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.734848 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:00.802105 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.802076 5090 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 01:20:00.806992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.806962 5090 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.807475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.807455 5090 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.819132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.819110 5090 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 01:20:00.819242 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.819154 5090 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.821418 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821392 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.821530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821428 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkc6\" (UniqueName: \"kubernetes.io/projected/72564aed-4e32-4815-8a3e-6f61a8c36e83-kube-api-access-lhkc6\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.821530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821449 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.821530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821509 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f35a7382-ce08-44b1-8427-33a5d30252b7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:20:00.821664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821542 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-env-overrides\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.821664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821572 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-netns\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.821664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821601 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssr7p\" (UniqueName: \"kubernetes.io/projected/5e6f6573-29b9-4b30-a82c-2b6e8948583f-kube-api-access-ssr7p\") pod \"network-check-source-6f78d59f99-dgzjj\" (UID: \"5e6f6573-29b9-4b30-a82c-2b6e8948583f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" Oct 31 01:20:00.821664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821626 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovn-node-metrics-cert\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.821664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821650 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-k8s-cni-cncf-io\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821747 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-conf-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821751 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f35a7382-ce08-44b1-8427-33a5d30252b7-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821778 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f35a7382-ce08-44b1-8427-33a5d30252b7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821806 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-env-overrides\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821830 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821837 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-multus\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821867 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-socket-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821904 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovn-node-metrics-cert\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.821926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821916 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-etc-kubernetes\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821951 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-trusted-ca\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.821967 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822025 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f35a7382-ce08-44b1-8427-33a5d30252b7-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822035 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e137e152-115f-4459-80fe-a92048049da2-agent-certs\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822070 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-device-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822114 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29151040-d2de-4a3c-b608-dc74c6aeee0e-serviceca\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822165 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-crio-socket\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822206 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.822301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822271 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8280d35c-b526-41e0-b4a2-0bd74b181789-serving-cert\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822336 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb74w\" (UniqueName: \"kubernetes.io/projected/3e5f8170-98cf-4487-84e2-0aeccb45ba6c-kube-api-access-bb74w\") pod \"volume-data-source-validator-58c9f5f46d-q2bv7\" (UID: \"3e5f8170-98cf-4487-84e2-0aeccb45ba6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822338 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-trusted-ca\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822384 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822437 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e137e152-115f-4459-80fe-a92048049da2-agent-certs\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822470 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8280d35c-b526-41e0-b4a2-0bd74b181789-serving-cert\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822506 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-etc-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822546 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822576 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.822649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822625 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-daemon-config\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822654 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8hh\" (UniqueName: \"kubernetes.io/projected/befed7f0-b23a-48a5-803e-6f8bf380012c-kube-api-access-fp8hh\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822679 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-conf\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822704 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-netns\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822749 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-cnibin\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822785 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822811 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822835 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822898 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9h6\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822940 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-host-slash\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822900 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-daemon-config\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.822984 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29151040-d2de-4a3c-b608-dc74c6aeee0e-serviceca\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823052 5090 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823073 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823096 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.823114 5090 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Oct 31 01:20:00.823150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823129 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-api-access-ndrdz\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823239 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-sys\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823292 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-tuned\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823317 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823416 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-slash\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823456 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-tuned\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823498 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823586 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e137e152-115f-4459-80fe-a92048049da2-konnectivity-ca\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823607 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxmkv\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823754 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysconfig\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823776 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e137e152-115f-4459-80fe-a92048049da2-konnectivity-ca\") pod \"konnectivity-agent-7rq4g\" (UID: \"e137e152-115f-4459-80fe-a92048049da2\") " pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:20:00.823792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823791 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-system-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823882 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-bin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823928 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pxs\" (UniqueName: \"kubernetes.io/projected/9fb8224e-3136-4261-8344-c4bc50dee44c-kube-api-access-s5pxs\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823959 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.823985 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tprw\" (UniqueName: \"kubernetes.io/projected/51211307-a4ca-4d06-95f7-02a7e5fef197-kube-api-access-7tprw\") pod \"migrator-66df5d496b-6km77\" (UID: \"51211307-a4ca-4d06-95f7-02a7e5fef197\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824009 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-var-lib-kubelet\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824035 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd88h\" (UniqueName: \"kubernetes.io/projected/8a3cb44a-89d0-4bb6-9815-3f52494b3047-kube-api-access-gd88h\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824061 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824336 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-registration-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824366 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a489a94c-56c7-45df-92cb-42177dadd9c1-cert\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824385 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-multus-certs\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824364 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824404 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:20:00.824496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824439 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824526 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824548 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a489a94c-56c7-45df-92cb-42177dadd9c1-cert\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824551 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shdj\" (UniqueName: \"kubernetes.io/projected/df649d60-b1cd-408d-a0fc-e74fd86e6896-kube-api-access-8shdj\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824591 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-stats-auth\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824610 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930232e3-20a2-4134-99ab-9817dede3186-tmp-dir\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824634 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97bgp\" (UniqueName: \"kubernetes.io/projected/930232e3-20a2-4134-99ab-9817dede3186-kube-api-access-97bgp\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824659 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5472b622-19ab-4582-a5ef-4f09314f0213-signing-cabundle\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824681 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-snapshots\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824692 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824709 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824731 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930232e3-20a2-4134-99ab-9817dede3186-tmp-dir\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824768 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-run\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824792 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-snapshots\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824825 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-stats-auth\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824827 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.824887 5090 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824892 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5472b622-19ab-4582-a5ef-4f09314f0213-signing-cabundle\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824916 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a3cb44a-89d0-4bb6-9815-3f52494b3047-config\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:20:00.824970 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.824956 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:01.324940736 +0000 UTC m=+1.705185937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.824994 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825064 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvkm\" (UniqueName: \"kubernetes.io/projected/5472b622-19ab-4582-a5ef-4f09314f0213-kube-api-access-rjvkm\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825123 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a3cb44a-89d0-4bb6-9815-3f52494b3047-config\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825212 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825243 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825272 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lxl\" (UniqueName: \"kubernetes.io/projected/5cf30068-f986-4286-b700-f68bdc0e5f75-kube-api-access-q5lxl\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825311 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825359 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825391 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825400 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825432 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e587446-856d-4580-a69f-94cd5953a81d-original-pull-secret\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825468 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825501 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825510 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df649d60-b1cd-408d-a0fc-e74fd86e6896-config-volume\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825530 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825541 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66jj\" (UniqueName: \"kubernetes.io/projected/4af660b2-97ac-4db7-b928-9d660b0c15a9-kube-api-access-z66jj\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825549 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825569 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-default-certificate\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825595 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825642 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/930232e3-20a2-4134-99ab-9817dede3186-hosts-file\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825669 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825673 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.825693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825695 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmsj\" (UniqueName: \"kubernetes.io/projected/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kube-api-access-9dmsj\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825758 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzdq\" (UniqueName: \"kubernetes.io/projected/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-kube-api-access-fdzdq\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825760 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df649d60-b1cd-408d-a0fc-e74fd86e6896-config-volume\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825780 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-default-certificate\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825872 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5472b622-19ab-4582-a5ef-4f09314f0213-signing-key\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825918 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb6xr\" (UniqueName: \"kubernetes.io/projected/8280d35c-b526-41e0-b4a2-0bd74b181789-kube-api-access-nb6xr\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825947 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cni-binary-copy\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.825990 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29151040-d2de-4a3c-b608-dc74c6aeee0e-host\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826029 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826071 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826091 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-kubelet\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826107 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-sys-fs\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826121 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-systemd-units\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826106 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5472b622-19ab-4582-a5ef-4f09314f0213-signing-key\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826144 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-node-log\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826167 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826185 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjkg8\" (UniqueName: \"kubernetes.io/projected/a489a94c-56c7-45df-92cb-42177dadd9c1-kube-api-access-vjkg8\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826200 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826211 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cni-binary-copy\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826286 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-iptables-alerter-script\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826330 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphc2\" (UniqueName: \"kubernetes.io/projected/43b11ef2-886e-4d57-9700-089f51fa360e-kube-api-access-xphc2\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826350 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df649d60-b1cd-408d-a0fc-e74fd86e6896-tmp-dir\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826366 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826382 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-script-lib\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826333 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4af660b2-97ac-4db7-b928-9d660b0c15a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826406 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtx7\" (UniqueName: \"kubernetes.io/projected/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6-kube-api-access-rdtx7\") pod \"migrator-66df5d496b-8tkh2\" (UID: \"6ae589ef-bc72-4b28-a05a-14dc01aa1dd6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826445 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgdm\" (UniqueName: \"kubernetes.io/projected/f35a7382-ce08-44b1-8427-33a5d30252b7-kube-api-access-rzgdm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826473 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df649d60-b1cd-408d-a0fc-e74fd86e6896-tmp-dir\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826522 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnjx7\" (UniqueName: \"kubernetes.io/projected/29151040-d2de-4a3c-b608-dc74c6aeee0e-kube-api-access-rnjx7\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826537 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-iptables-alerter-script\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826555 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-host\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826542 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.826706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826674 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-config\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826778 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cnibin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826806 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd04b736-45e6-44cc-a032-e47ef93641a9-config\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826814 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-script-lib\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826807 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826842 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-netd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826871 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkt44\" (UniqueName: \"kubernetes.io/projected/dd04b736-45e6-44cc-a032-e47ef93641a9-kube-api-access-hkt44\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826888 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbbj\" (UniqueName: \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826910 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826937 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-systemd\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826960 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-lib-modules\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.826983 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-system-cni-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827019 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-data-volume\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827043 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827112 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827114 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-data-volume\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827140 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-bin\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827180 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-tmp\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827184 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827213 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827229 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-var-lib-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827252 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827272 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3cb44a-89d0-4bb6-9815-3f52494b3047-serving-cert\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827250 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8280d35c-b526-41e0-b4a2-0bd74b181789-tmp\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827296 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-systemd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827352 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-ovn\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827409 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-log-socket\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827457 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-service-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827500 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827530 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-tmp\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827596 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5cf30068-f986-4286-b700-f68bdc0e5f75-tmp\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827616 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-kubelet\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827644 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a3cb44a-89d0-4bb6-9815-3f52494b3047-serving-cert\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827654 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdsp\" (UniqueName: \"kubernetes.io/projected/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-kube-api-access-4zdsp\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827666 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8280d35c-b526-41e0-b4a2-0bd74b181789-service-ca-bundle\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827694 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-config\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827768 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd04b736-45e6-44cc-a032-e47ef93641a9-serving-cert\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827791 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827898 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-dbus\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827941 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827977 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.827984 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd04b736-45e6-44cc-a032-e47ef93641a9-serving-cert\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828004 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-kubernetes\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828031 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/72564aed-4e32-4815-8a3e-6f61a8c36e83-ovnkube-config\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828032 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-socket-dir-parent\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828057 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828066 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828098 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-hostroot\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828126 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.828161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828167 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-nginx-conf\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:00.829463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828199 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-kubelet-config\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.829463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828208 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.829463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.828226 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-modprobe-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.835304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.835285 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkc6\" (UniqueName: \"kubernetes.io/projected/72564aed-4e32-4815-8a3e-6f61a8c36e83-kube-api-access-lhkc6\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.835398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.835380 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssr7p\" (UniqueName: \"kubernetes.io/projected/5e6f6573-29b9-4b30-a82c-2b6e8948583f-kube-api-access-ssr7p\") pod \"network-check-source-6f78d59f99-dgzjj\" (UID: \"5e6f6573-29b9-4b30-a82c-2b6e8948583f\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dgzjj" Oct 31 01:20:00.835913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.835899 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb74w\" (UniqueName: \"kubernetes.io/projected/3e5f8170-98cf-4487-84e2-0aeccb45ba6c-kube-api-access-bb74w\") pod \"volume-data-source-validator-58c9f5f46d-q2bv7\" (UID: \"3e5f8170-98cf-4487-84e2-0aeccb45ba6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-q2bv7" Oct 31 01:20:00.841279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.841262 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrdz\" (UniqueName: \"kubernetes.io/projected/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-api-access-ndrdz\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.841492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.841478 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.843566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.843550 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55prj\" (UniqueName: \"kubernetes.io/projected/578616cb-95f4-49ef-afec-65f65cdb2f71-kube-api-access-55prj\") pod \"network-check-target-klvct\" (UID: \"578616cb-95f4-49ef-afec-65f65cdb2f71\") " pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:20:00.844363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.844349 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9h6\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.844539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.844525 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tprw\" (UniqueName: \"kubernetes.io/projected/51211307-a4ca-4d06-95f7-02a7e5fef197-kube-api-access-7tprw\") pod \"migrator-66df5d496b-6km77\" (UID: \"51211307-a4ca-4d06-95f7-02a7e5fef197\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6km77" Oct 31 01:20:00.845847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.845831 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd88h\" (UniqueName: \"kubernetes.io/projected/8a3cb44a-89d0-4bb6-9815-3f52494b3047-kube-api-access-gd88h\") pod \"service-ca-operator-544d749c8c-fvkhr\" (UID: \"8a3cb44a-89d0-4bb6-9815-3f52494b3047\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" Oct 31 01:20:00.845959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.845944 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8hh\" (UniqueName: \"kubernetes.io/projected/befed7f0-b23a-48a5-803e-6f8bf380012c-kube-api-access-fp8hh\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.846553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.846538 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxmkv\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.847370 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.847355 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pxs\" (UniqueName: \"kubernetes.io/projected/9fb8224e-3136-4261-8344-c4bc50dee44c-kube-api-access-s5pxs\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:00.851925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.851908 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphc2\" (UniqueName: \"kubernetes.io/projected/43b11ef2-886e-4d57-9700-089f51fa360e-kube-api-access-xphc2\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:20:00.852576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.852559 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97bgp\" (UniqueName: \"kubernetes.io/projected/930232e3-20a2-4134-99ab-9817dede3186-kube-api-access-97bgp\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:20:00.853265 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.853244 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdsp\" (UniqueName: \"kubernetes.io/projected/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-kube-api-access-4zdsp\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.853491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.853477 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtx7\" (UniqueName: \"kubernetes.io/projected/6ae589ef-bc72-4b28-a05a-14dc01aa1dd6-kube-api-access-rdtx7\") pod \"migrator-66df5d496b-8tkh2\" (UID: \"6ae589ef-bc72-4b28-a05a-14dc01aa1dd6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8tkh2" Oct 31 01:20:00.854652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.854631 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmsj\" (UniqueName: \"kubernetes.io/projected/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kube-api-access-9dmsj\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.856025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.856006 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvkm\" (UniqueName: \"kubernetes.io/projected/5472b622-19ab-4582-a5ef-4f09314f0213-kube-api-access-rjvkm\") pod \"service-ca-5fffc97bdf-bw6g8\" (UID: \"5472b622-19ab-4582-a5ef-4f09314f0213\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bw6g8" Oct 31 01:20:00.856744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.856728 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzdq\" (UniqueName: \"kubernetes.io/projected/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-kube-api-access-fdzdq\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:20:00.858371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.858345 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shdj\" (UniqueName: \"kubernetes.io/projected/df649d60-b1cd-408d-a0fc-e74fd86e6896-kube-api-access-8shdj\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.858533 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.858517 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.858741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.858704 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkt44\" (UniqueName: \"kubernetes.io/projected/dd04b736-45e6-44cc-a032-e47ef93641a9-kube-api-access-hkt44\") pod \"console-operator-7cd56b8dbc-zkw9h\" (UID: \"dd04b736-45e6-44cc-a032-e47ef93641a9\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:00.858822 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.858807 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb6xr\" (UniqueName: \"kubernetes.io/projected/8280d35c-b526-41e0-b4a2-0bd74b181789-kube-api-access-nb6xr\") pod \"insights-operator-7c4986d449-7hl7l\" (UID: \"8280d35c-b526-41e0-b4a2-0bd74b181789\") " pod="openshift-insights/insights-operator-7c4986d449-7hl7l" Oct 31 01:20:00.859870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.859853 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnjx7\" (UniqueName: \"kubernetes.io/projected/29151040-d2de-4a3c-b608-dc74c6aeee0e-kube-api-access-rnjx7\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:20:00.860052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.860038 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgdm\" (UniqueName: \"kubernetes.io/projected/f35a7382-ce08-44b1-8427-33a5d30252b7-kube-api-access-rzgdm\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2s452\" (UID: \"f35a7382-ce08-44b1-8427-33a5d30252b7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" Oct 31 01:20:00.861142 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.861123 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66jj\" (UniqueName: \"kubernetes.io/projected/4af660b2-97ac-4db7-b928-9d660b0c15a9-kube-api-access-z66jj\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.861693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.861675 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbbj\" (UniqueName: \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\") pod \"periodic-gathering-slnps-lhfzm\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:20:00.862043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.862027 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjkg8\" (UniqueName: \"kubernetes.io/projected/a489a94c-56c7-45df-92cb-42177dadd9c1-kube-api-access-vjkg8\") pod \"ingress-canary-jn8kf\" (UID: \"a489a94c-56c7-45df-92cb-42177dadd9c1\") " pod="openshift-ingress-canary/ingress-canary-jn8kf" Oct 31 01:20:00.864483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.864456 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lxl\" (UniqueName: \"kubernetes.io/projected/5cf30068-f986-4286-b700-f68bdc0e5f75-kube-api-access-q5lxl\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.928648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928621 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-registration-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.928648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928650 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-multus-certs\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.928837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928670 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:00.928837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928685 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-run\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.928837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928759 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-registration-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.928837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928763 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-multus-certs\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.928837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928782 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-run\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.928837 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.928809 5090 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:20:00.928837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928828 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928857 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.928870 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:20:01.428853517 +0000 UTC m=+1.809098718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928911 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928899 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928934 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/930232e3-20a2-4134-99ab-9817dede3186-hosts-file\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928970 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928969 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.928996 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29151040-d2de-4a3c-b608-dc74c6aeee0e-host\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929033 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/930232e3-20a2-4134-99ab-9817dede3186-hosts-file\") pod \"node-resolver-mz75q\" (UID: \"930232e3-20a2-4134-99ab-9817dede3186\") " pod="openshift-dns/node-resolver-mz75q" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.929034 5090 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.929055 5090 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929067 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.929089 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:01.429074482 +0000 UTC m=+1.809319688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929044 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29151040-d2de-4a3c-b608-dc74c6aeee0e-host\") pod \"node-ca-pmc4f\" (UID: \"29151040-d2de-4a3c-b608-dc74c6aeee0e\") " pod="openshift-image-registry/node-ca-pmc4f" Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.929145 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:01.429129425 +0000 UTC m=+1.809374626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:20:00.929169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929169 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929194 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-kubelet\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929222 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-sys-fs\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929232 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-kubelet\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929244 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-systemd-units\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929286 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-systemd-units\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929293 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-sys-fs\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929340 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-node-log\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.929372 5090 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929377 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.929412 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:20:01.429399162 +0000 UTC m=+1.809644364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929430 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-host\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929445 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929461 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-node-log\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929467 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cnibin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929510 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-cnibin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929516 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-netd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929515 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929517 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-host\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929571 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-netd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929611 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-systemd\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929631 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-lib-modules\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929654 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-system-cni-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929680 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-bin\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929707 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-var-lib-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929754 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929772 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-systemd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929786 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-ovn\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929802 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-log-socket\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929799 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-cni-bin\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929813 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-var-lib-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929830 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-ovn-kubernetes\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929847 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-log-socket\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929852 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-system-cni-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929851 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-systemd\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929857 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-run-ovn\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929867 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-lib-modules\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929876 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929893 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-systemd\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929903 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-kubelet\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929922 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-dbus\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929938 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-kubernetes\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929981 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-socket-dir-parent\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.929991 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-kubelet\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930005 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-kubernetes\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930009 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930016 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-dbus\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930033 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-hostroot\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930044 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-socket-dir-parent\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930058 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-kubelet-config\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930073 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930083 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-modprobe-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930086 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-hostroot\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930105 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930115 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e587446-856d-4580-a69f-94cd5953a81d-kubelet-config\") pod \"global-pull-secret-syncer-c5d98\" (UID: \"7e587446-856d-4580-a69f-94cd5953a81d\") " pod="kube-system/global-pull-secret-syncer-c5d98" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930134 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-netns\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930151 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930157 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-k8s-cni-cncf-io\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.930123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930174 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-modprobe-d\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930182 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-conf-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930189 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-netns\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930196 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-run-k8s-cni-cncf-io\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930205 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-multus\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930208 5090 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930225 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-socket-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930227 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-multus-conf-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930241 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-etc-kubernetes\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930264 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930287 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-multus\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930290 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-device-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930308 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-etc-kubernetes\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930311 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-socket-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930327 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-crio-socket\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930334 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-device-dir\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930351 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930379 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-crio-socket\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930378 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930411 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-etc-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930436 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930463 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-conf\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930500 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-etc-openvswitch\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930516 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930507 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-netns\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930546 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-run-netns\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930558 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-cnibin\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930582 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930605 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-host-slash\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930631 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-sys\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930657 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-slash\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930666 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4af660b2-97ac-4db7-b928-9d660b0c15a9-cnibin\") pod \"multus-additional-cni-plugins-h7k55\" (UID: \"4af660b2-97ac-4db7-b928-9d660b0c15a9\") " pod="openshift-multus/multus-additional-cni-plugins-h7k55" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930701 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysconfig\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930708 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72564aed-4e32-4815-8a3e-6f61a8c36e83-host-slash\") pod \"ovnkube-node-kjrt4\" (UID: \"72564aed-4e32-4815-8a3e-6f61a8c36e83\") " pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:00.930735 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:01.430697778 +0000 UTC m=+1.810943013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930767 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-system-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930796 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-bin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930804 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysconfig\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930822 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930849 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-var-lib-kubelet\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930853 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ffd51cb1-85bf-472e-b0d7-14b6bf1cf591-host-slash\") pod \"iptables-alerter-fcqc8\" (UID: \"ffd51cb1-85bf-472e-b0d7-14b6bf1cf591\") " pod="openshift-network-operator/iptables-alerter-fcqc8" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930633 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-etc-sysctl-conf\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.930857 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-sys\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.931106 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-system-cni-dir\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.931144 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dfa8037e-1fcd-4d2f-b8d5-edc99be74be9-host-var-lib-cni-bin\") pod \"multus-xfqs2\" (UID: \"dfa8037e-1fcd-4d2f-b8d5-edc99be74be9\") " pod="openshift-multus/multus-xfqs2" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.931190 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wlcps\" (UID: \"fa6ce2e8-64af-45dd-a942-7bbb99bfdb2c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wlcps" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.931233 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf30068-f986-4286-b700-f68bdc0e5f75-var-lib-kubelet\") pod \"tuned-bf87z\" (UID: \"5cf30068-f986-4286-b700-f68bdc0e5f75\") " pod="openshift-cluster-node-tuning-operator/tuned-bf87z" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.932934 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43b11ef2-886e-4d57-9700-089f51fa360e-metrics-certs\") pod \"network-metrics-daemon-wff52\" (UID: \"43b11ef2-886e-4d57-9700-089f51fa360e\") " pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.932962 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"image-registry-fb9bcc569-v7h6k\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.932997 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mz75q_930232e3-20a2-4134-99ab-9817dede3186/dns-node-resolver/0.log" Oct 31 01:20:00.933285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.933111 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"image-registry-54d575cd44-5jzvn\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:00.935761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:00.934105 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df649d60-b1cd-408d-a0fc-e74fd86e6896-metrics-tls\") pod \"dns-default-mrqzl\" (UID: \"df649d60-b1cd-408d-a0fc-e74fd86e6896\") " pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:01.003760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.003738 5090 scope.go:122] "RemoveContainer" containerID="176f3b933e5201fea082acd69c7789423a6693b3fe23407901e56699c4c6cc14" Oct 31 01:20:01.004137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.004122 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:01.006232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.006213 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wff52" Oct 31 01:20:01.009612 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.009593 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:01.020105 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.020083 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:01.057525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.057493 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.197925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.197870 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.318922 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.318835 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54d575cd44-5jzvn"] Oct 31 01:20:01.322691 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:01.322649 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb123e4c4_57cd_49f8_985b_158b4deab82d.slice/crio-36fd7396d483de17e3aa60d2d15e0777522e4008a805d7828f67576dc8bafd2d WatchSource:0}: Error finding container 36fd7396d483de17e3aa60d2d15e0777522e4008a805d7828f67576dc8bafd2d: Status 404 returned error can't find the container with id 36fd7396d483de17e3aa60d2d15e0777522e4008a805d7828f67576dc8bafd2d Oct 31 01:20:01.336828 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.336801 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:01.336977 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.336957 5090 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:01.337033 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.337018 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.337001951 +0000 UTC m=+2.717247152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:01.354229 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.354195 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.364971 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.364947 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wff52"] Oct 31 01:20:01.373076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.373057 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:20:01.406297 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.406263 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrqzl"] Oct 31 01:20:01.410586 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:01.410559 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf649d60_b1cd_408d_a0fc_e74fd86e6896.slice/crio-0c8bbe96df532e6619f276d0ab50579ab2b1a7fb8cae347b1c657d2fbe7882e3 WatchSource:0}: Error finding container 0c8bbe96df532e6619f276d0ab50579ab2b1a7fb8cae347b1c657d2fbe7882e3: Status 404 returned error can't find the container with id 0c8bbe96df532e6619f276d0ab50579ab2b1a7fb8cae347b1c657d2fbe7882e3 Oct 31 01:20:01.437354 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.437321 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:01.437506 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.437373 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:01.437506 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.437413 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:01.437506 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.437436 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:01.437506 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.437464 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437510 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.437488514 +0000 UTC m=+2.817733734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437547 5090 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437583 5090 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437592 5090 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437608 5090 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437610 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.437591593 +0000 UTC m=+2.817836807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437637 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.437625347 +0000 UTC m=+2.817870554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437653 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.437644698 +0000 UTC m=+2.817889899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:20:01.437741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:01.437676 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:20:02.437667641 +0000 UTC m=+2.817912881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:20:01.439239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.439174 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fb9bcc569-v7h6k"] Oct 31 01:20:01.445864 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:01.445833 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd51c16f_f327_4939_ad47_4acc6aaf62c2.slice/crio-45905ae4041ee7f7137db32bdd8fffabf961b6682f9b445b769b3b48752b953b WatchSource:0}: Error finding container 45905ae4041ee7f7137db32bdd8fffabf961b6682f9b445b769b3b48752b953b: Status 404 returned error can't find the container with id 45905ae4041ee7f7137db32bdd8fffabf961b6682f9b445b769b3b48752b953b Oct 31 01:20:01.499098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.499072 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.649075 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.649037 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.784739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.784682 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.811642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.811606 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" event={"ID":"bd51c16f-f327-4939-ad47-4acc6aaf62c2","Type":"ContainerStarted","Data":"d9bc075d468b09ce36c828d6edc08b3c6c0d0c8aad302bb1f8809ca96bd38dbb"} Oct 31 01:20:01.811838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.811662 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" event={"ID":"bd51c16f-f327-4939-ad47-4acc6aaf62c2","Type":"ContainerStarted","Data":"45905ae4041ee7f7137db32bdd8fffabf961b6682f9b445b769b3b48752b953b"} Oct 31 01:20:01.813069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.813040 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wff52" event={"ID":"43b11ef2-886e-4d57-9700-089f51fa360e","Type":"ContainerStarted","Data":"be94a0f914efe20c8361552f2705aed5f1998b42738d4e5cb04680f64ce8486b"} Oct 31 01:20:01.814589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.814561 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" event={"ID":"b123e4c4-57cd-49f8-985b-158b4deab82d","Type":"ContainerStarted","Data":"9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224"} Oct 31 01:20:01.814704 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.814621 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" event={"ID":"b123e4c4-57cd-49f8-985b-158b4deab82d","Type":"ContainerStarted","Data":"36fd7396d483de17e3aa60d2d15e0777522e4008a805d7828f67576dc8bafd2d"} Oct 31 01:20:01.816578 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.816559 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:20:01.816676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.816658 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" event={"ID":"dd04b736-45e6-44cc-a032-e47ef93641a9","Type":"ContainerStarted","Data":"73541f86a02a397d75c30d8148a4cc9b6e3402a70e3643e1f13fbb4ea0205d89"} Oct 31 01:20:01.818079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.818054 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrqzl" event={"ID":"df649d60-b1cd-408d-a0fc-e74fd86e6896","Type":"ContainerStarted","Data":"0c8bbe96df532e6619f276d0ab50579ab2b1a7fb8cae347b1c657d2fbe7882e3"} Oct 31 01:20:01.928997 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.928929 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:01.931952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:01.931931 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pmc4f_29151040-d2de-4a3c-b608-dc74c6aeee0e/node-ca/0.log" Oct 31 01:20:02.077114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.077083 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:02.296575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.296541 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:02.346807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.346768 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:02.346978 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.346940 5090 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:02.347031 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.347024 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.347003204 +0000 UTC m=+4.727248418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:02.448075 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.448026 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:02.448257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.448100 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:02.448257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.448134 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:02.448257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.448166 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:02.448478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.448257 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:02.448478 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.448399 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.448386326 +0000 UTC m=+4.828631540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:20:02.448809 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.448779 5090 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:20:02.448882 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.448839 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.448823364 +0000 UTC m=+4.829068579 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:20:02.448932 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.448894 5090 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:20:02.448932 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.448928 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.448913828 +0000 UTC m=+4.829159043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:20:02.449029 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.448980 5090 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:20:02.449029 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.449011 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.449001263 +0000 UTC m=+4.829246478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:20:02.449108 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.449062 5090 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:20:02.449108 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:02.449090 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:20:04.449081074 +0000 UTC m=+4.829326289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:20:02.538407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.538374 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jn8kf_a489a94c-56c7-45df-92cb-42177dadd9c1/serve-healthcheck-canary/0.log" Oct 31 01:20:02.610823 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.610623 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:02.683773 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.683687 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:02.733539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.733511 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6km77_51211307-a4ca-4d06-95f7-02a7e5fef197/migrator/0.log" Oct 31 01:20:02.820920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.820887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:02.824688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.824667 5090 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 01:20:02.933543 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.933516 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-8tkh2_6ae589ef-bc72-4b28-a05a-14dc01aa1dd6/migrator/0.log" Oct 31 01:20:02.941613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:02.941594 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:03.012584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.012549 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.134275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.134194 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2s452_f35a7382-ce08-44b1-8427-33a5d30252b7/kube-storage-version-migrator-operator/0.log" Oct 31 01:20:03.152164 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.152131 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.335923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.335879 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.533129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.533096 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:03.534818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.534788 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:03.831014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.830920 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wff52" event={"ID":"43b11ef2-886e-4d57-9700-089f51fa360e","Type":"ContainerStarted","Data":"5a72247c3a61cb169457d9c43461778ffabad26013c358f60d2984fa9636dc80"} Oct 31 01:20:03.831014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.830956 5090 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 01:20:03.831014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.830988 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wff52" event={"ID":"43b11ef2-886e-4d57-9700-089f51fa360e","Type":"ContainerStarted","Data":"bf297e0732bcb052af2ef7b852626106051254586f03de919db94fc2dbe00e55"} Oct 31 01:20:03.943246 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:03.943215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:04.368227 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.368144 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:04.368379 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.368313 5090 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:04.368421 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.368383 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls podName:6b85121a-782e-4f5a-b984-a775a806b231 nodeName:}" failed. No retries permitted until 2026-10-31 01:20:08.36836641 +0000 UTC m=+8.748611648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-8wdsg" (UID: "6b85121a-782e-4f5a-b984-a775a806b231") : secret "cluster-monitoring-operator-tls" not found Oct 31 01:20:04.469236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.469199 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:04.469385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.469255 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:04.469385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.469293 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:04.469385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.469310 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:04.469385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.469339 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:04.469385 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469363 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:08.469340195 +0000 UTC m=+8.849585409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : configmap references non-existent config key: service-ca.crt Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469421 5090 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469421 5090 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469439 5090 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469423 5090 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469470 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls podName:cb9da61d-3af1-4aa1-8ddd-de96b78ffafe nodeName:}" failed. No retries permitted until 2026-10-31 01:20:08.469454647 +0000 UTC m=+8.849699852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8jv29" (UID: "cb9da61d-3af1-4aa1-8ddd-de96b78ffafe") : secret "insights-runtime-extractor-tls" not found Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469492 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls podName:9fb8224e-3136-4261-8344-c4bc50dee44c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:08.469475843 +0000 UTC m=+8.849721072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-wlpvf" (UID: "9fb8224e-3136-4261-8344-c4bc50dee44c") : secret "samples-operator-tls" not found Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469523 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert podName:3bf687ff-9445-49b0-804a-5eb5ca7ddedc nodeName:}" failed. No retries permitted until 2026-10-31 01:20:08.469513659 +0000 UTC m=+8.849758861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-d74bn" (UID: "3bf687ff-9445-49b0-804a-5eb5ca7ddedc") : secret "networking-console-plugin-cert" not found Oct 31 01:20:04.469598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:04.469539 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs podName:befed7f0-b23a-48a5-803e-6f8bf380012c nodeName:}" failed. No retries permitted until 2026-10-31 01:20:08.469530363 +0000 UTC m=+8.849775569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs") pod "router-default-54786d95-fr8g9" (UID: "befed7f0-b23a-48a5-803e-6f8bf380012c") : secret "router-metrics-certs-default" not found Oct 31 01:20:04.476329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.476308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:04.606889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.606855 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:04.748437 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.748407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:04.836435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.836400 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrqzl" event={"ID":"df649d60-b1cd-408d-a0fc-e74fd86e6896","Type":"ContainerStarted","Data":"d9b1f60684f27e0743384abb643a3c780233136aeb38b69f54bacaae0e53c35f"} Oct 31 01:20:04.836920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.836477 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrqzl" event={"ID":"df649d60-b1cd-408d-a0fc-e74fd86e6896","Type":"ContainerStarted","Data":"737e19f7a60b9407c19e3e66de0647736f3e5134992523cd1efeb63b4784a39d"} Oct 31 01:20:04.836920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.836514 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:04.838174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.838153 5090 generic.go:356] "Generic (PLEG): container finished" podID="8a3cb44a-89d0-4bb6-9815-3f52494b3047" containerID="89f8d684f504ecc87d8b032fc2912a6406a7377c86a2224f4196f88d958a9be3" exitCode=0 Oct 31 01:20:04.838240 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.838225 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" event={"ID":"8a3cb44a-89d0-4bb6-9815-3f52494b3047","Type":"ContainerDied","Data":"89f8d684f504ecc87d8b032fc2912a6406a7377c86a2224f4196f88d958a9be3"} Oct 31 01:20:04.838601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:04.838588 5090 scope.go:122] "RemoveContainer" containerID="89f8d684f504ecc87d8b032fc2912a6406a7377c86a2224f4196f88d958a9be3" Oct 31 01:20:05.142024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:05.141984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:05.334435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:05.334381 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:05.531842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:05.531806 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:05.814318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:05.814220 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:05.845265 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:05.845232 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-fvkhr" event={"ID":"8a3cb44a-89d0-4bb6-9815-3f52494b3047","Type":"ContainerStarted","Data":"91fdb00fb21938b1e12e485938e0cb81f60e59ed2b24776330e6b19be44fbc92"} Oct 31 01:20:05.940282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:05.940248 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:20:05.953880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:05.953855 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:06.736617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:06.736578 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:06.878336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:06.878308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:07.134216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:07.134182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:07.321769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:07.321736 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:07.531249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:07.531214 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:07.734370 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:07.734326 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:07.736299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:07.736277 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:07.736452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:07.736438 5090 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 01:20:07.744221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:07.744202 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zkw9h" Oct 31 01:20:08.132288 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.132243 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:08.411488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.411397 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:08.414047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.414018 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b85121a-782e-4f5a-b984-a775a806b231-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-8wdsg\" (UID: \"6b85121a-782e-4f5a-b984-a775a806b231\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:08.500999 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.500968 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" Oct 31 01:20:08.512242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.512216 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:08.512401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.512282 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:08.512401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.512315 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:08.512401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.512342 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:08.512401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.512374 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:08.513166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.513130 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/befed7f0-b23a-48a5-803e-6f8bf380012c-service-ca-bundle\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:08.515522 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.515500 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/befed7f0-b23a-48a5-803e-6f8bf380012c-metrics-certs\") pod \"router-default-54786d95-fr8g9\" (UID: \"befed7f0-b23a-48a5-803e-6f8bf380012c\") " pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:08.515813 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.515788 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cb9da61d-3af1-4aa1-8ddd-de96b78ffafe-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jv29\" (UID: \"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe\") " pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:08.516313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.516288 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3bf687ff-9445-49b0-804a-5eb5ca7ddedc-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-d74bn\" (UID: \"3bf687ff-9445-49b0-804a-5eb5ca7ddedc\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:08.516398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.516287 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9fb8224e-3136-4261-8344-c4bc50dee44c-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-wlpvf\" (UID: \"9fb8224e-3136-4261-8344-c4bc50dee44c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:08.522113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.522093 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jv29" Oct 31 01:20:08.775835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.775798 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg"] Oct 31 01:20:08.779593 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:08.779569 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b85121a_782e_4f5a_b984_a775a806b231.slice/crio-6508bee2c73fbfc65509335ac72effbf98249413a1002f7a48597e2efe1e94b2 WatchSource:0}: Error finding container 6508bee2c73fbfc65509335ac72effbf98249413a1002f7a48597e2efe1e94b2: Status 404 returned error can't find the container with id 6508bee2c73fbfc65509335ac72effbf98249413a1002f7a48597e2efe1e94b2 Oct 31 01:20:08.803180 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.803157 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:08.803833 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.803814 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" Oct 31 01:20:08.805109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.804877 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" Oct 31 01:20:08.805244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.805217 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jv29"] Oct 31 01:20:08.810785 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:08.810757 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9da61d_3af1_4aa1_8ddd_de96b78ffafe.slice/crio-555836e290876bc5f20232cb7d322bcbc00d35e2b1994a887d0b4d5d1890fd89 WatchSource:0}: Error finding container 555836e290876bc5f20232cb7d322bcbc00d35e2b1994a887d0b4d5d1890fd89: Status 404 returned error can't find the container with id 555836e290876bc5f20232cb7d322bcbc00d35e2b1994a887d0b4d5d1890fd89 Oct 31 01:20:08.862915 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.862877 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" event={"ID":"6b85121a-782e-4f5a-b984-a775a806b231","Type":"ContainerStarted","Data":"6508bee2c73fbfc65509335ac72effbf98249413a1002f7a48597e2efe1e94b2"} Oct 31 01:20:08.864961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:08.864933 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jv29" event={"ID":"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe","Type":"ContainerStarted","Data":"555836e290876bc5f20232cb7d322bcbc00d35e2b1994a887d0b4d5d1890fd89"} Oct 31 01:20:09.101908 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.101837 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-d74bn"] Oct 31 01:20:09.105235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.105202 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf"] Oct 31 01:20:09.106001 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:09.105964 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf687ff_9445_49b0_804a_5eb5ca7ddedc.slice/crio-687b5652d0d1368bade635571839a2f3d0a73b9aeda0918731205fd77625c8e2 WatchSource:0}: Error finding container 687b5652d0d1368bade635571839a2f3d0a73b9aeda0918731205fd77625c8e2: Status 404 returned error can't find the container with id 687b5652d0d1368bade635571839a2f3d0a73b9aeda0918731205fd77625c8e2 Oct 31 01:20:09.109085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.109066 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54786d95-fr8g9"] Oct 31 01:20:09.113904 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:09.113878 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbefed7f0_b23a_48a5_803e_6f8bf380012c.slice/crio-845c13492d251bd1f2ed826486276056eeee56f5899651ab4d67a077a97a9606 WatchSource:0}: Error finding container 845c13492d251bd1f2ed826486276056eeee56f5899651ab4d67a077a97a9606: Status 404 returned error can't find the container with id 845c13492d251bd1f2ed826486276056eeee56f5899651ab4d67a077a97a9606 Oct 31 01:20:09.122429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.122407 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:09.355374 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.355304 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kjrt4" Oct 31 01:20:09.870927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.870889 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jv29" event={"ID":"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe","Type":"ContainerStarted","Data":"78d2c5658a627783d8e0fbf151e975385f2fd7320c5e85e09b926e1863dfd530"} Oct 31 01:20:09.873255 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.873221 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" event={"ID":"3bf687ff-9445-49b0-804a-5eb5ca7ddedc","Type":"ContainerStarted","Data":"687b5652d0d1368bade635571839a2f3d0a73b9aeda0918731205fd77625c8e2"} Oct 31 01:20:09.875113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.875079 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54786d95-fr8g9" event={"ID":"befed7f0-b23a-48a5-803e-6f8bf380012c","Type":"ContainerStarted","Data":"89598aa209d673984eee4bd6b72328c881ec8a707b5d1fd032de2a32635f6de7"} Oct 31 01:20:09.875230 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.875129 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54786d95-fr8g9" event={"ID":"befed7f0-b23a-48a5-803e-6f8bf380012c","Type":"ContainerStarted","Data":"845c13492d251bd1f2ed826486276056eeee56f5899651ab4d67a077a97a9606"} Oct 31 01:20:09.878886 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:09.878860 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" event={"ID":"9fb8224e-3136-4261-8344-c4bc50dee44c","Type":"ContainerStarted","Data":"d1b42f865b10911a4a11bc947395e1345f1a057909e1daaafa17e3724318c283"} Oct 31 01:20:10.420376 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.420308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:10.560690 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.560658 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:10.803981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.803930 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:10.807982 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.807954 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:10.886954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.886916 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jv29" event={"ID":"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe","Type":"ContainerStarted","Data":"d54536fccd2436b3efe64ba40f25a05f5622332840bb285268259477fa2aed1e"} Oct 31 01:20:10.890760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.890692 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:10.892012 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.891989 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54786d95-fr8g9" Oct 31 01:20:10.893329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:10.893303 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-8wdsg" event={"ID":"6b85121a-782e-4f5a-b984-a775a806b231","Type":"ContainerStarted","Data":"fec2c8c17167a84f81ca11964a19a2f9fd991ff3977bb147957a6740aae60c4d"} Oct 31 01:20:11.060097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:11.060065 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:11.204184 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:11.204155 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:11.352911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:11.352836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:11.544167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:11.544130 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:11.898544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:11.898498 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-d74bn" event={"ID":"3bf687ff-9445-49b0-804a-5eb5ca7ddedc","Type":"ContainerStarted","Data":"ff918f9e9467ff3655d0e71e2597b0587105f22b50c3ce1f86cbb2da47ab598e"} Oct 31 01:20:12.906743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:12.906636 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jv29" event={"ID":"cb9da61d-3af1-4aa1-8ddd-de96b78ffafe","Type":"ContainerStarted","Data":"9ae621298f4a46712626c8e11a43f0be526dd4b7aaa23485f2ac0265eb365d02"} Oct 31 01:20:13.017049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:13.017015 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:13.165430 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:13.165364 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:13.912221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:13.912137 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" event={"ID":"9fb8224e-3136-4261-8344-c4bc50dee44c","Type":"ContainerStarted","Data":"dafceee9dc0bebc36a243a29ac0e24f93b2807345f12ef2816c05d30651da9cb"} Oct 31 01:20:13.912221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:13.912191 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-wlpvf" event={"ID":"9fb8224e-3136-4261-8344-c4bc50dee44c","Type":"ContainerStarted","Data":"9f31d4443045ed7ee2e4e70760375092729c6de25d797bcaa5c3a9535bcb5e0f"} Oct 31 01:20:14.848153 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:14.848127 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mrqzl" Oct 31 01:20:15.305238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:15.305210 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7rq4g" Oct 31 01:20:15.825368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:15.825331 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:15.958001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:15.957963 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:16.358124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.358080 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-s29ln"] Oct 31 01:20:16.361648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.361597 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.365167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.365144 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7469b5bd86-d46tv"] Oct 31 01:20:16.368425 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.368403 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq"] Oct 31 01:20:16.368818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.368797 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.371408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.371390 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" Oct 31 01:20:16.379112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.379092 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-n9pz7\"" Oct 31 01:20:16.379724 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.379694 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 01:20:16.380027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.380007 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 01:20:16.388359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.388337 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 01:20:16.389015 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.388989 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 01:20:16.389192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.389178 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 01:20:16.389833 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.389813 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-s29ln"] Oct 31 01:20:16.392321 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.392301 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 01:20:16.392473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.392458 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq"] Oct 31 01:20:16.398847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.398824 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7469b5bd86-d46tv"] Oct 31 01:20:16.414014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.413990 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54d575cd44-5jzvn"] Oct 31 01:20:16.418861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.418834 5090 patch_prober.go:29] interesting pod/image-registry-54d575cd44-5jzvn container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 01:20:16.418975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.418879 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" podUID="b123e4c4-57cd-49f8-985b-158b4deab82d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:20:16.477911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.477884 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 01:20:16.478082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.478006 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9q6j8\"" Oct 31 01:20:16.478082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.478021 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kxnhs\"" Oct 31 01:20:16.478082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.478018 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 01:20:16.478195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.478085 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 01:20:16.478511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.478493 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-674c48bf5d-rzkcr"] Oct 31 01:20:16.481766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.481748 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.485517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485500 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-config\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.485608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485531 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-serving-cert\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.485608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485554 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a-tmp\") pod \"downloads-6dd789c8b7-s29ln\" (UID: \"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a\") " pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.485608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485604 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-service-ca\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.485744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485669 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjv9h\" (UniqueName: \"kubernetes.io/projected/394b8516-d37f-45dd-8a79-db1b2e9a3af7-kube-api-access-zjv9h\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.485785 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485773 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c32161dd-88f4-42b2-a8dc-fc2ffdd504e4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bnsfq\" (UID: \"c32161dd-88f4-42b2-a8dc-fc2ffdd504e4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" Oct 31 01:20:16.485840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485806 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-oauth-config\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.485900 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485887 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774lg\" (UniqueName: \"kubernetes.io/projected/e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a-kube-api-access-774lg\") pod \"downloads-6dd789c8b7-s29ln\" (UID: \"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a\") " pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.485938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485906 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394b8516-d37f-45dd-8a79-db1b2e9a3af7-tmp\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.485938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485922 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-oauth-serving-cert\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.486013 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.485942 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-trusted-ca-bundle\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.498195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.498169 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674c48bf5d-rzkcr"] Oct 31 01:20:16.528913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.528884 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:16.587249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587178 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7779b35b-c98d-4220-9d81-1f44d9440fae-registry-certificates\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.587249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587208 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7779b35b-c98d-4220-9d81-1f44d9440fae-installation-pull-secrets\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.587249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587244 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774lg\" (UniqueName: \"kubernetes.io/projected/e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a-kube-api-access-774lg\") pod \"downloads-6dd789c8b7-s29ln\" (UID: \"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a\") " pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.587447 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587287 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394b8516-d37f-45dd-8a79-db1b2e9a3af7-tmp\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.587447 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587362 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-oauth-serving-cert\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.587514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587448 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-trusted-ca-bundle\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.587514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587474 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7779b35b-c98d-4220-9d81-1f44d9440fae-image-registry-private-configuration\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.587514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587503 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7779b35b-c98d-4220-9d81-1f44d9440fae-ca-trust-extracted\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.587619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587530 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnl9\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-kube-api-access-6lnl9\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.587665 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587644 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-config\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.587699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587682 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394b8516-d37f-45dd-8a79-db1b2e9a3af7-tmp\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.587759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587690 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7779b35b-c98d-4220-9d81-1f44d9440fae-trusted-ca\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.587802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587767 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-serving-cert\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.587864 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587798 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-registry-tls\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.587914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587872 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a-tmp\") pod \"downloads-6dd789c8b7-s29ln\" (UID: \"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a\") " pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.587964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587917 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-service-ca\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.587964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587950 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjv9h\" (UniqueName: \"kubernetes.io/projected/394b8516-d37f-45dd-8a79-db1b2e9a3af7-kube-api-access-zjv9h\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.588058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.587991 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c32161dd-88f4-42b2-a8dc-fc2ffdd504e4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bnsfq\" (UID: \"c32161dd-88f4-42b2-a8dc-fc2ffdd504e4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" Oct 31 01:20:16.588058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.588052 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-oauth-config\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.588151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.588080 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-bound-sa-token\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.588151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.588127 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-oauth-serving-cert\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.588246 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.588211 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-trusted-ca-bundle\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.588302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.588271 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a-tmp\") pod \"downloads-6dd789c8b7-s29ln\" (UID: \"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a\") " pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.588413 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.588358 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-config\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.588882 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.588806 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-service-ca\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.590448 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.590415 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-serving-cert\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.590543 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.590516 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-oauth-config\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.590594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.590546 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c32161dd-88f4-42b2-a8dc-fc2ffdd504e4-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-bnsfq\" (UID: \"c32161dd-88f4-42b2-a8dc-fc2ffdd504e4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" Oct 31 01:20:16.599054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.599036 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774lg\" (UniqueName: \"kubernetes.io/projected/e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a-kube-api-access-774lg\") pod \"downloads-6dd789c8b7-s29ln\" (UID: \"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a\") " pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.599146 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.599120 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjv9h\" (UniqueName: \"kubernetes.io/projected/394b8516-d37f-45dd-8a79-db1b2e9a3af7-kube-api-access-zjv9h\") pod \"console-7469b5bd86-d46tv\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.675427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.675394 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:20:16.680202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.680179 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:16.684190 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.684171 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:16.689450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689429 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7779b35b-c98d-4220-9d81-1f44d9440fae-trusted-ca\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.689557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689473 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-registry-tls\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.689557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689543 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-bound-sa-token\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.689660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689572 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7779b35b-c98d-4220-9d81-1f44d9440fae-registry-certificates\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.689660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689599 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7779b35b-c98d-4220-9d81-1f44d9440fae-installation-pull-secrets\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.689660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689653 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7779b35b-c98d-4220-9d81-1f44d9440fae-image-registry-private-configuration\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.689828 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689679 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7779b35b-c98d-4220-9d81-1f44d9440fae-ca-trust-extracted\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.689828 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.689704 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnl9\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-kube-api-access-6lnl9\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.690442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.690420 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7779b35b-c98d-4220-9d81-1f44d9440fae-ca-trust-extracted\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.690627 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.690600 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7779b35b-c98d-4220-9d81-1f44d9440fae-trusted-ca\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.691231 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.691189 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7779b35b-c98d-4220-9d81-1f44d9440fae-registry-certificates\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.692932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.692909 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-registry-tls\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.693080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.693063 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" Oct 31 01:20:16.693754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.693705 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7779b35b-c98d-4220-9d81-1f44d9440fae-installation-pull-secrets\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.693913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.693892 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7779b35b-c98d-4220-9d81-1f44d9440fae-image-registry-private-configuration\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.702558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.702520 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnl9\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-kube-api-access-6lnl9\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.702702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.702608 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7779b35b-c98d-4220-9d81-1f44d9440fae-bound-sa-token\") pod \"image-registry-674c48bf5d-rzkcr\" (UID: \"7779b35b-c98d-4220-9d81-1f44d9440fae\") " pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.793398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.793374 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:16.960281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.960252 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-s29ln"] Oct 31 01:20:16.963829 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:16.963801 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fe2a1c_6e57_426e_acc0_8f18c1e2bb8a.slice/crio-d86b87e98fba59c0c0089ec3022886af9152d13e00148495cf6942bb1ae34a42 WatchSource:0}: Error finding container d86b87e98fba59c0c0089ec3022886af9152d13e00148495cf6942bb1ae34a42: Status 404 returned error can't find the container with id d86b87e98fba59c0c0089ec3022886af9152d13e00148495cf6942bb1ae34a42 Oct 31 01:20:16.971072 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.970317 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7469b5bd86-d46tv"] Oct 31 01:20:16.977050 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:16.977023 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394b8516_d37f_45dd_8a79_db1b2e9a3af7.slice/crio-e3644505dba44482e21a108061eb059716589a2f9d0b4d6bc0fa677e48d5cd47 WatchSource:0}: Error finding container e3644505dba44482e21a108061eb059716589a2f9d0b4d6bc0fa677e48d5cd47: Status 404 returned error can't find the container with id e3644505dba44482e21a108061eb059716589a2f9d0b4d6bc0fa677e48d5cd47 Oct 31 01:20:16.988386 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:16.988363 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq"] Oct 31 01:20:16.991667 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:16.991496 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc32161dd_88f4_42b2_a8dc_fc2ffdd504e4.slice/crio-87c1431659e4ba102f85dd9c6c5d3890b9df3357e7d07a00946cc407dfc7a2e3 WatchSource:0}: Error finding container 87c1431659e4ba102f85dd9c6c5d3890b9df3357e7d07a00946cc407dfc7a2e3: Status 404 returned error can't find the container with id 87c1431659e4ba102f85dd9c6c5d3890b9df3357e7d07a00946cc407dfc7a2e3 Oct 31 01:20:17.086270 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.086240 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-674c48bf5d-rzkcr"] Oct 31 01:20:17.089700 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:17.089660 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7779b35b_c98d_4220_9d81_1f44d9440fae.slice/crio-e8f9894c1502b6182f96b3526c462af939fc92145dcb515c46ff631be73dd8db WatchSource:0}: Error finding container e8f9894c1502b6182f96b3526c462af939fc92145dcb515c46ff631be73dd8db: Status 404 returned error can't find the container with id e8f9894c1502b6182f96b3526c462af939fc92145dcb515c46ff631be73dd8db Oct 31 01:20:17.931699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.931617 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469b5bd86-d46tv" event={"ID":"394b8516-d37f-45dd-8a79-db1b2e9a3af7","Type":"ContainerStarted","Data":"e3644505dba44482e21a108061eb059716589a2f9d0b4d6bc0fa677e48d5cd47"} Oct 31 01:20:17.933268 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.933213 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" event={"ID":"c32161dd-88f4-42b2-a8dc-fc2ffdd504e4","Type":"ContainerStarted","Data":"87c1431659e4ba102f85dd9c6c5d3890b9df3357e7d07a00946cc407dfc7a2e3"} Oct 31 01:20:17.936312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.935438 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" event={"ID":"7779b35b-c98d-4220-9d81-1f44d9440fae","Type":"ContainerStarted","Data":"a5c2f86e7622f7230940dd2a0e107b12622a24644d37c681c53ece70d32b0c87"} Oct 31 01:20:17.936312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.935482 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" event={"ID":"7779b35b-c98d-4220-9d81-1f44d9440fae","Type":"ContainerStarted","Data":"e8f9894c1502b6182f96b3526c462af939fc92145dcb515c46ff631be73dd8db"} Oct 31 01:20:17.936312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.936241 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:17.937764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.937737 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-s29ln" event={"ID":"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a","Type":"ContainerStarted","Data":"d86b87e98fba59c0c0089ec3022886af9152d13e00148495cf6942bb1ae34a42"} Oct 31 01:20:17.960140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:17.959892 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" podStartSLOduration=1.959872922 podStartE2EDuration="1.959872922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:17.958499315 +0000 UTC m=+18.338744539" watchObservedRunningTime="2026-10-31 01:20:17.959872922 +0000 UTC m=+18.340118147" Oct 31 01:20:18.943988 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:18.943955 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" event={"ID":"c32161dd-88f4-42b2-a8dc-fc2ffdd504e4","Type":"ContainerStarted","Data":"99783bebace356c5d08c8e29d87b35954621300239fcefe823c658417cdce4fa"} Oct 31 01:20:18.960327 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:18.960284 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" podStartSLOduration=1.5233858869999999 podStartE2EDuration="2.960271592s" totalImagesPullingTime="1.436885705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:18.959834577 +0000 UTC m=+19.340079801" watchObservedRunningTime="2026-10-31 01:20:18.960271592 +0000 UTC m=+19.340516809" Oct 31 01:20:19.476629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:19.476598 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:19.615653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:19.615619 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:19.948266 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:19.948236 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" Oct 31 01:20:19.954290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:19.954269 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-bnsfq" Oct 31 01:20:20.314051 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:20.314012 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:20.469916 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:20.469881 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:21.029358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:21.029323 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:21.958898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:21.958855 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469b5bd86-d46tv" event={"ID":"394b8516-d37f-45dd-8a79-db1b2e9a3af7","Type":"ContainerStarted","Data":"9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211"} Oct 31 01:20:21.978989 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:21.978947 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7469b5bd86-d46tv" podStartSLOduration=1.6593768770000001 podStartE2EDuration="5.97893421s" totalImagesPullingTime="4.319557333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:21.977796286 +0000 UTC m=+22.358041519" watchObservedRunningTime="2026-10-31 01:20:21.97893421 +0000 UTC m=+22.359179485" Oct 31 01:20:22.829080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:22.829046 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:22.967585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:22.967544 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:23.149931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:23.149852 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:23.286575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:23.286535 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:23.835841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:23.835814 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:20:24.217338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.217265 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:24.357445 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.357379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:24.509627 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.509591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:24.668981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.668948 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:24.703291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.701481 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wknwn"] Oct 31 01:20:24.733639 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.733614 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769176 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-wtmp\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769234 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769310 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-sys\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769333 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-tls\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769362 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-root\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769472 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-textfile\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769504 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2749cd14-51e0-4df3-b410-8357b589b1ad-metrics-client-ca\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769528 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbkhv\" (UniqueName: \"kubernetes.io/projected/2749cd14-51e0-4df3-b410-8357b589b1ad-kube-api-access-bbkhv\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.769634 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.769614 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.786501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.786472 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 01:20:24.786683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.786617 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 01:20:24.786683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.786657 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-44nrr\"" Oct 31 01:20:24.786859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.786825 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 01:20:24.787086 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.786948 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 01:20:24.821910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.821877 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:24.870014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.869973 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870025 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-wtmp\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870063 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870099 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-sys\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870124 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-tls\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870159 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-root\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870191 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-sys\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870209 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-root\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870228 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-wtmp\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870222 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-textfile\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870303 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2749cd14-51e0-4df3-b410-8357b589b1ad-metrics-client-ca\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870331 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbkhv\" (UniqueName: \"kubernetes.io/projected/2749cd14-51e0-4df3-b410-8357b589b1ad-kube-api-access-bbkhv\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870947 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870509 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-textfile\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.870947 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870666 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-accelerators-collector-config\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.871022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.870945 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2749cd14-51e0-4df3-b410-8357b589b1ad-metrics-client-ca\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.872892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.872868 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.873035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.873020 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2749cd14-51e0-4df3-b410-8357b589b1ad-node-exporter-tls\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.882256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.882233 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbkhv\" (UniqueName: \"kubernetes.io/projected/2749cd14-51e0-4df3-b410-8357b589b1ad-kube-api-access-bbkhv\") pod \"node-exporter-wknwn\" (UID: \"2749cd14-51e0-4df3-b410-8357b589b1ad\") " pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:24.959062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:24.959028 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.098649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:25.098573 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wknwn" Oct 31 01:20:25.133522 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:25.133487 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2749cd14_51e0_4df3_b410_8357b589b1ad.slice/crio-e10eba7cce28bb06035d199ed387131c58bf6ae61307ac0f48cbaec100500d92 WatchSource:0}: Error finding container e10eba7cce28bb06035d199ed387131c58bf6ae61307ac0f48cbaec100500d92: Status 404 returned error can't find the container with id e10eba7cce28bb06035d199ed387131c58bf6ae61307ac0f48cbaec100500d92 Oct 31 01:20:25.135515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:25.135486 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.275300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:25.275266 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.804014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:25.803979 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:25.981293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:25.981254 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wknwn" event={"ID":"2749cd14-51e0-4df3-b410-8357b589b1ad","Type":"ContainerStarted","Data":"e10eba7cce28bb06035d199ed387131c58bf6ae61307ac0f48cbaec100500d92"} Oct 31 01:20:26.002694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.002657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:26.137365 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.137291 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:26.420541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.420153 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:26.685426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.685348 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:26.685426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.685419 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:26.690762 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.690701 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:26.986380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.986353 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wknwn" event={"ID":"2749cd14-51e0-4df3-b410-8357b589b1ad","Type":"ContainerStarted","Data":"cc002b5ffad5ca1ad49c59070ad6a345b9a9c9a8af1aa3594a37cdd618cb9c15"} Oct 31 01:20:26.990250 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:26.990231 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:20:27.412521 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:27.412490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:27.545157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:27.545126 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:27.754954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:27.754927 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:27.940477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:27.940448 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:27.992843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:27.992810 5090 generic.go:356] "Generic (PLEG): container finished" podID="2749cd14-51e0-4df3-b410-8357b589b1ad" containerID="cc002b5ffad5ca1ad49c59070ad6a345b9a9c9a8af1aa3594a37cdd618cb9c15" exitCode=0 Oct 31 01:20:27.993314 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:27.992903 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wknwn" event={"ID":"2749cd14-51e0-4df3-b410-8357b589b1ad","Type":"ContainerDied","Data":"cc002b5ffad5ca1ad49c59070ad6a345b9a9c9a8af1aa3594a37cdd618cb9c15"} Oct 31 01:20:28.087420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:28.087390 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:28.999654 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:28.999614 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wknwn" event={"ID":"2749cd14-51e0-4df3-b410-8357b589b1ad","Type":"ContainerStarted","Data":"7a872a90b9982b829666e15bd37e4c1f331877035d8a2d98fac51c2c3981e657"} Oct 31 01:20:29.000090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:28.999664 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wknwn" event={"ID":"2749cd14-51e0-4df3-b410-8357b589b1ad","Type":"ContainerStarted","Data":"b409b12a8bf5cc8345de9781f11da4f8703da5af40445f60dd2607e4f2ccfeb6"} Oct 31 01:20:29.020109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:29.020063 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wknwn" podStartSLOduration=3.182556963 podStartE2EDuration="5.020051185s" totalImagesPullingTime="1.743590216s" totalInitContainerRuntime="93.904006ms" isStatefulPod=true podCreationTimestamp="2026-10-31 01:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:29.019591737 +0000 UTC m=+29.399836959" watchObservedRunningTime="2026-10-31 01:20:29.020051185 +0000 UTC m=+29.400296407" Oct 31 01:20:29.437738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:29.437613 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:29.575151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:29.575116 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:29.709913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:29.709836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:31.062375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:31.062341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:33.299824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:33.299793 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:33.432541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:33.432509 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:34.559304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.559270 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f9ddc9998-pqcjg"] Oct 31 01:20:34.562176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.562159 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.574777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.574753 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9ddc9998-pqcjg"] Oct 31 01:20:34.656842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.656813 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-tmp\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.656991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.656848 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-service-ca\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.656991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.656866 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-config\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.656991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.656931 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-trusted-ca-bundle\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.656991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.656961 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-oauth-serving-cert\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.657257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.657036 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-serving-cert\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.657257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.657059 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbg46\" (UniqueName: \"kubernetes.io/projected/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-kube-api-access-rbg46\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.657257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.657086 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-oauth-config\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758163 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-serving-cert\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758205 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbg46\" (UniqueName: \"kubernetes.io/projected/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-kube-api-access-rbg46\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758240 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-oauth-config\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758318 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-tmp\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758340 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-service-ca\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758364 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-config\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758392 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-trusted-ca-bundle\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.758563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.758509 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-oauth-serving-cert\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.759185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.759156 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-tmp\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.759302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.759275 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-config\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.759752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.759735 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-oauth-serving-cert\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.759858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.759836 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-service-ca\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.759906 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.759848 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-trusted-ca-bundle\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.760826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.760803 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-oauth-config\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.760993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.760979 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-serving-cert\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.770107 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.770086 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbg46\" (UniqueName: \"kubernetes.io/projected/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-kube-api-access-rbg46\") pod \"console-7f9ddc9998-pqcjg\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:34.788170 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:34.788145 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:20:34.872693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:34.872615 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:35.150416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:35.150182 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f9ddc9998-pqcjg"] Oct 31 01:20:35.153246 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:20:35.153221 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ad36ca_1f47_4a1f_8a95_8b71ebbf4f0e.slice/crio-eacef9fbe43f6c0ff33667d101ede15b8fb7e131f676e282a36ae8cc6ec630e8 WatchSource:0}: Error finding container eacef9fbe43f6c0ff33667d101ede15b8fb7e131f676e282a36ae8cc6ec630e8: Status 404 returned error can't find the container with id eacef9fbe43f6c0ff33667d101ede15b8fb7e131f676e282a36ae8cc6ec630e8 Oct 31 01:20:35.943151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:35.943122 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-klvct" Oct 31 01:20:36.031210 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:36.031177 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9ddc9998-pqcjg" event={"ID":"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e","Type":"ContainerStarted","Data":"b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f"} Oct 31 01:20:36.031348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:36.031228 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9ddc9998-pqcjg" event={"ID":"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e","Type":"ContainerStarted","Data":"eacef9fbe43f6c0ff33667d101ede15b8fb7e131f676e282a36ae8cc6ec630e8"} Oct 31 01:20:36.051207 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:36.051152 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f9ddc9998-pqcjg" podStartSLOduration=2.05113446 podStartE2EDuration="2.05113446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:36.050567142 +0000 UTC m=+36.430812384" watchObservedRunningTime="2026-10-31 01:20:36.05113446 +0000 UTC m=+36.431379685" Oct 31 01:20:37.065557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:37.065524 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:37.199648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:37.199610 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:38.910677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:38.910649 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:39.043350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:39.043314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:39.952514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:39.952486 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-674c48bf5d-rzkcr" Oct 31 01:20:40.008044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:40.008015 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb9bcc569-v7h6k"] Oct 31 01:20:40.214066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:40.213953 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:40.367223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:40.367191 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:41.434743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.434684 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" podUID="b123e4c4-57cd-49f8-985b-158b4deab82d" containerName="registry" containerID="cri-o://9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224" gracePeriod=30 Oct 31 01:20:41.687990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.687940 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:41.821251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821222 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.821424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821325 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.821424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821369 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.821424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821400 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.821602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821446 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.821602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821478 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.821602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821504 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.821602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.821566 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\") pod \"b123e4c4-57cd-49f8-985b-158b4deab82d\" (UID: \"b123e4c4-57cd-49f8-985b-158b4deab82d\") " Oct 31 01:20:41.822483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.822404 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:41.822611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.822505 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:20:41.825463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.825411 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:41.826513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.826491 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:41.826618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.826544 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:41.826676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.826627 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:20:41.826676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.826659 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "kube-api-access-dxmkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:20:41.831449 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.831422 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted" pod "b123e4c4-57cd-49f8-985b-158b4deab82d" (UID: "b123e4c4-57cd-49f8-985b-158b4deab82d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:20:41.922933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922901 5090 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b123e4c4-57cd-49f8-985b-158b4deab82d-ca-trust-extracted\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:41.922933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922926 5090 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:41.922933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922936 5090 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-image-registry-private-configuration\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:41.923134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922946 5090 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-bound-sa-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:41.923134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922956 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxmkv\" (UniqueName: \"kubernetes.io/projected/b123e4c4-57cd-49f8-985b-158b4deab82d-kube-api-access-dxmkv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:41.923134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922965 5090 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b123e4c4-57cd-49f8-985b-158b4deab82d-installation-pull-secrets\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:41.923134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922975 5090 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-registry-certificates\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:41.923134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:41.922985 5090 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b123e4c4-57cd-49f8-985b-158b4deab82d-trusted-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:20:42.058051 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.058012 5090 generic.go:356] "Generic (PLEG): container finished" podID="b123e4c4-57cd-49f8-985b-158b4deab82d" containerID="9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224" exitCode=0 Oct 31 01:20:42.058212 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.058062 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" Oct 31 01:20:42.058212 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.058070 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" event={"ID":"b123e4c4-57cd-49f8-985b-158b4deab82d","Type":"ContainerDied","Data":"9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224"} Oct 31 01:20:42.058212 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.058117 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54d575cd44-5jzvn" event={"ID":"b123e4c4-57cd-49f8-985b-158b4deab82d","Type":"ContainerDied","Data":"36fd7396d483de17e3aa60d2d15e0777522e4008a805d7828f67576dc8bafd2d"} Oct 31 01:20:42.058212 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.058138 5090 scope.go:122] "RemoveContainer" containerID="9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224" Oct 31 01:20:42.067963 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.067947 5090 scope.go:122] "RemoveContainer" containerID="9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224" Oct 31 01:20:42.068253 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:20:42.068234 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224\": container with ID starting with 9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224 not found: ID does not exist" containerID="9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224" Oct 31 01:20:42.068291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.068263 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224"} err="failed to get container status \"9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224\": rpc error: code = NotFound desc = could not find container \"9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224\": container with ID starting with 9460c4ae430fca70b08bc7d3f2ff35ecc4f63477c0b0afa5989b6e7ac9bb2224 not found: ID does not exist" Oct 31 01:20:42.080867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.080844 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54d575cd44-5jzvn"] Oct 31 01:20:42.088591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:42.088567 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54d575cd44-5jzvn"] Oct 31 01:20:43.747911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:43.745297 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b123e4c4-57cd-49f8-985b-158b4deab82d" path="/var/lib/kubelet/pods/b123e4c4-57cd-49f8-985b-158b4deab82d/volumes" Oct 31 01:20:44.872852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:44.872821 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:44.873212 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:44.872888 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:44.877611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:44.877592 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:45.076377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:45.076346 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:20:45.142318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:45.142248 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7469b5bd86-d46tv"] Oct 31 01:20:46.215347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:46.215308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:46.357949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:46.357920 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:48.517248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:48.517192 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:48.721157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:48.721126 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:49.477194 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:49.477150 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:49.617554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:49.617523 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:20:52.810839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:20:52.810809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:04.511877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:04.511842 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:04.657317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:04.657282 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:04.786698 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:21:04.786613 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:21:05.030091 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.030049 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" podUID="bd51c16f-f327-4939-ad47-4acc6aaf62c2" containerName="registry" containerID="cri-o://d9bc075d468b09ce36c828d6edc08b3c6c0d0c8aad302bb1f8809ca96bd38dbb" gracePeriod=30 Oct 31 01:21:05.157881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.157840 5090 generic.go:356] "Generic (PLEG): container finished" podID="bd51c16f-f327-4939-ad47-4acc6aaf62c2" containerID="d9bc075d468b09ce36c828d6edc08b3c6c0d0c8aad302bb1f8809ca96bd38dbb" exitCode=0 Oct 31 01:21:05.158018 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.157903 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" event={"ID":"bd51c16f-f327-4939-ad47-4acc6aaf62c2","Type":"ContainerDied","Data":"d9bc075d468b09ce36c828d6edc08b3c6c0d0c8aad302bb1f8809ca96bd38dbb"} Oct 31 01:21:05.274597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.274576 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:21:05.336138 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336072 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336138 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336107 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336138 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336139 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336161 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336187 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336214 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336240 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336322 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\") pod \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\" (UID: \"bd51c16f-f327-4939-ad47-4acc6aaf62c2\") " Oct 31 01:21:05.336887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.336835 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:05.337299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.337265 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:05.338820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.338789 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:05.339163 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.339136 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:05.339297 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.339277 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:05.339559 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.339531 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "kube-api-access-cs9h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:05.339605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.339568 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:05.343391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.343371 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted" pod "bd51c16f-f327-4939-ad47-4acc6aaf62c2" (UID: "bd51c16f-f327-4939-ad47-4acc6aaf62c2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:05.437217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437183 5090 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd51c16f-f327-4939-ad47-4acc6aaf62c2-ca-trust-extracted\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.437217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437214 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs9h6\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-kube-api-access-cs9h6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.437217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437225 5090 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-installation-pull-secrets\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.437485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437236 5090 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-bound-sa-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.437485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437245 5090 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.437485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437254 5090 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-registry-certificates\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.437485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437265 5090 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd51c16f-f327-4939-ad47-4acc6aaf62c2-trusted-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:05.437485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:05.437278 5090 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd51c16f-f327-4939-ad47-4acc6aaf62c2-image-registry-private-configuration\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:06.163913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:06.163820 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" event={"ID":"bd51c16f-f327-4939-ad47-4acc6aaf62c2","Type":"ContainerDied","Data":"45905ae4041ee7f7137db32bdd8fffabf961b6682f9b445b769b3b48752b953b"} Oct 31 01:21:06.163913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:06.163863 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fb9bcc569-v7h6k" Oct 31 01:21:06.163913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:06.163887 5090 scope.go:122] "RemoveContainer" containerID="d9bc075d468b09ce36c828d6edc08b3c6c0d0c8aad302bb1f8809ca96bd38dbb" Oct 31 01:21:06.183166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:06.183142 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fb9bcc569-v7h6k"] Oct 31 01:21:06.190205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:06.190186 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fb9bcc569-v7h6k"] Oct 31 01:21:07.747002 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:07.746970 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd51c16f-f327-4939-ad47-4acc6aaf62c2" path="/var/lib/kubelet/pods/bd51c16f-f327-4939-ad47-4acc6aaf62c2/volumes" Oct 31 01:21:10.169238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.169188 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7469b5bd86-d46tv" podUID="394b8516-d37f-45dd-8a79-db1b2e9a3af7" containerName="console" containerID="cri-o://9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211" gracePeriod=15 Oct 31 01:21:10.444835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.444806 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:21:10.574095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574051 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/394b8516-d37f-45dd-8a79-db1b2e9a3af7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/394b8516-d37f-45dd-8a79-db1b2e9a3af7-tmp\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574124 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-oauth-config\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574168 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-trusted-ca-bundle\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574207 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-serving-cert\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574240 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394b8516-d37f-45dd-8a79-db1b2e9a3af7-kube-api-access-zjv9h\" (UniqueName: \"kubernetes.io/projected/394b8516-d37f-45dd-8a79-db1b2e9a3af7-kube-api-access-zjv9h\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574314 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-config\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-config\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574351 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-service-ca\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-service-ca\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574379 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-oauth-serving-cert\") pod \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\" (UID: \"394b8516-d37f-45dd-8a79-db1b2e9a3af7\") " Oct 31 01:21:10.574814 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574684 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-trusted-ca-bundle" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:10.574957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574933 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-oauth-serving-cert" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:10.574957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574821 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-config" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:10.575085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.574973 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-service-ca" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:10.575085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.575036 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/394b8516-d37f-45dd-8a79-db1b2e9a3af7-tmp" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:10.577049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.577013 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-serving-cert" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:10.577160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.577117 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-oauth-config" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:21:10.577205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.577169 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394b8516-d37f-45dd-8a79-db1b2e9a3af7-kube-api-access-zjv9h" pod "394b8516-d37f-45dd-8a79-db1b2e9a3af7" (UID: "394b8516-d37f-45dd-8a79-db1b2e9a3af7"). InnerVolumeSpecName "kube-api-access-zjv9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:10.675751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675660 5090 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:10.675751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675692 5090 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-serving-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:10.675751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675709 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjv9h\" (UniqueName: \"kubernetes.io/projected/394b8516-d37f-45dd-8a79-db1b2e9a3af7-kube-api-access-zjv9h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:10.675751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675744 5090 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:10.675751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675753 5090 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:10.676053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675761 5090 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/394b8516-d37f-45dd-8a79-db1b2e9a3af7-oauth-serving-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:10.676053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675770 5090 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/394b8516-d37f-45dd-8a79-db1b2e9a3af7-tmp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:10.676053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:10.675783 5090 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/394b8516-d37f-45dd-8a79-db1b2e9a3af7-console-oauth-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:11.190482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.190444 5090 generic.go:356] "Generic (PLEG): container finished" podID="394b8516-d37f-45dd-8a79-db1b2e9a3af7" containerID="9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211" exitCode=0 Oct 31 01:21:11.190972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.190498 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7469b5bd86-d46tv" Oct 31 01:21:11.190972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.190516 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469b5bd86-d46tv" event={"ID":"394b8516-d37f-45dd-8a79-db1b2e9a3af7","Type":"ContainerDied","Data":"9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211"} Oct 31 01:21:11.190972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.190580 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7469b5bd86-d46tv" event={"ID":"394b8516-d37f-45dd-8a79-db1b2e9a3af7","Type":"ContainerDied","Data":"e3644505dba44482e21a108061eb059716589a2f9d0b4d6bc0fa677e48d5cd47"} Oct 31 01:21:11.190972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.190608 5090 scope.go:122] "RemoveContainer" containerID="9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211" Oct 31 01:21:11.201035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.201006 5090 scope.go:122] "RemoveContainer" containerID="9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211" Oct 31 01:21:11.201296 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:21:11.201274 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211\": container with ID starting with 9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211 not found: ID does not exist" containerID="9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211" Oct 31 01:21:11.201387 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.201310 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211"} err="failed to get container status \"9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211\": rpc error: code = NotFound desc = could not find container \"9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211\": container with ID starting with 9a836c6bcc6bee832c90cc08d1e9e5891a94a2eb165945d9987be27987687211 not found: ID does not exist" Oct 31 01:21:11.212176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.212154 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7469b5bd86-d46tv"] Oct 31 01:21:11.221192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.216762 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7469b5bd86-d46tv"] Oct 31 01:21:11.358747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.358686 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:11.502203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.502167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:11.745411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:11.745377 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394b8516-d37f-45dd-8a79-db1b2e9a3af7" path="/var/lib/kubelet/pods/394b8516-d37f-45dd-8a79-db1b2e9a3af7/volumes" Oct 31 01:21:13.201570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:13.201531 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:13.341690 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:13.341657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:15.214127 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:15.214023 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-s29ln" event={"ID":"e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a","Type":"ContainerStarted","Data":"c24876485b475f5595f4c4872d802ec300a953c558327dfd3f3ea897786c86d9"} Oct 31 01:21:15.214622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:15.214209 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:21:15.233260 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:15.233207 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-s29ln" podStartSLOduration=1.37202289 podStartE2EDuration="59.233190198s" totalImagesPullingTime="57.861167308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:15.232764816 +0000 UTC m=+75.613010062" watchObservedRunningTime="2026-10-31 01:21:15.233190198 +0000 UTC m=+75.613435423" Oct 31 01:21:15.235498 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:15.235476 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-s29ln" Oct 31 01:21:19.236592 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:19.236555 5090 generic.go:356] "Generic (PLEG): container finished" podID="9e634f3d-0490-480a-9484-a1ee415c5fb1" containerID="06a2c9cfe6606ac01e54ec86918d53b44968543e1f8af1dd095348dff99b3108" exitCode=0 Oct 31 01:21:19.237087 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:19.236637 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slnps-lhfzm" event={"ID":"9e634f3d-0490-480a-9484-a1ee415c5fb1","Type":"ContainerDied","Data":"06a2c9cfe6606ac01e54ec86918d53b44968543e1f8af1dd095348dff99b3108"} Oct 31 01:21:19.488003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:19.487911 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:19.621821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:19.621769 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:20.423204 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.423176 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:21:20.464524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.464489 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\") pod \"9e634f3d-0490-480a-9484-a1ee415c5fb1\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " Oct 31 01:21:20.464703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.464551 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\" (UniqueName: \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\") pod \"9e634f3d-0490-480a-9484-a1ee415c5fb1\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " Oct 31 01:21:20.464703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.464689 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\") pod \"9e634f3d-0490-480a-9484-a1ee415c5fb1\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " Oct 31 01:21:20.464866 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.464751 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\") pod \"9e634f3d-0490-480a-9484-a1ee415c5fb1\" (UID: \"9e634f3d-0490-480a-9484-a1ee415c5fb1\") " Oct 31 01:21:20.465100 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.465072 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle" pod "9e634f3d-0490-480a-9484-a1ee415c5fb1" (UID: "9e634f3d-0490-480a-9484-a1ee415c5fb1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:20.465323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.465222 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle" pod "9e634f3d-0490-480a-9484-a1ee415c5fb1" (UID: "9e634f3d-0490-480a-9484-a1ee415c5fb1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:21:20.465323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.465258 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path" pod "9e634f3d-0490-480a-9484-a1ee415c5fb1" (UID: "9e634f3d-0490-480a-9484-a1ee415c5fb1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:21:20.467393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.467368 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj" pod "9e634f3d-0490-480a-9484-a1ee415c5fb1" (UID: "9e634f3d-0490-480a-9484-a1ee415c5fb1"). InnerVolumeSpecName "kube-api-access-rvbbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:21:20.566362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.566327 5090 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:20.566362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.566364 5090 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e634f3d-0490-480a-9484-a1ee415c5fb1-archives-path\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:20.566539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.566378 5090 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e634f3d-0490-480a-9484-a1ee415c5fb1-service-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:20.566539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:20.566388 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbbj\" (UniqueName: \"kubernetes.io/projected/9e634f3d-0490-480a-9484-a1ee415c5fb1-kube-api-access-rvbbj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:21:21.254396 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:21.254358 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-slnps-lhfzm" event={"ID":"9e634f3d-0490-480a-9484-a1ee415c5fb1","Type":"ContainerDied","Data":"f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d"} Oct 31 01:21:21.254593 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:21.254429 5090 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4ba8108e6f24bc09b2d558d7596d3eb1ea64dee8e50e394bdfa8b27bea8ed5d" Oct 31 01:21:21.254593 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:21.254492 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-slnps-lhfzm" Oct 31 01:21:21.729839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:21.729745 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:21.871875 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:21.871843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:25.273047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.273014 5090 generic.go:356] "Generic (PLEG): container finished" podID="f35a7382-ce08-44b1-8427-33a5d30252b7" containerID="72160e323b3c04c7aed02102262f85a6c6e1a950632c2f92588e55b577874275" exitCode=0 Oct 31 01:21:25.273514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.273090 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" event={"ID":"f35a7382-ce08-44b1-8427-33a5d30252b7","Type":"ContainerDied","Data":"72160e323b3c04c7aed02102262f85a6c6e1a950632c2f92588e55b577874275"} Oct 31 01:21:25.273514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.273437 5090 scope.go:122] "RemoveContainer" containerID="72160e323b3c04c7aed02102262f85a6c6e1a950632c2f92588e55b577874275" Oct 31 01:21:25.274772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.274752 5090 generic.go:356] "Generic (PLEG): container finished" podID="8280d35c-b526-41e0-b4a2-0bd74b181789" containerID="ab058ea00b533dbe69a0edec4719238d620cb771d874d2b2f2374f747aa7703f" exitCode=0 Oct 31 01:21:25.274847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.274822 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7hl7l" event={"ID":"8280d35c-b526-41e0-b4a2-0bd74b181789","Type":"ContainerDied","Data":"ab058ea00b533dbe69a0edec4719238d620cb771d874d2b2f2374f747aa7703f"} Oct 31 01:21:25.275083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.275071 5090 scope.go:122] "RemoveContainer" containerID="ab058ea00b533dbe69a0edec4719238d620cb771d874d2b2f2374f747aa7703f" Oct 31 01:21:25.752666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.752629 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-x9h8v-jmvqw"] Oct 31 01:21:25.753080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753060 5090 cpu_manager.go:414] "containerMap: removing container" podUID="9e634f3d-0490-480a-9484-a1ee415c5fb1" containerName="insights-gathering" Oct 31 01:21:25.753159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753082 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e634f3d-0490-480a-9484-a1ee415c5fb1" containerName="insights-gathering" Oct 31 01:21:25.753159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753096 5090 cpu_manager.go:414] "containerMap: removing container" podUID="b123e4c4-57cd-49f8-985b-158b4deab82d" containerName="registry" Oct 31 01:21:25.753159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753104 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b123e4c4-57cd-49f8-985b-158b4deab82d" containerName="registry" Oct 31 01:21:25.753159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753128 5090 cpu_manager.go:414] "containerMap: removing container" podUID="bd51c16f-f327-4939-ad47-4acc6aaf62c2" containerName="registry" Oct 31 01:21:25.753159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753135 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd51c16f-f327-4939-ad47-4acc6aaf62c2" containerName="registry" Oct 31 01:21:25.753159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753153 5090 cpu_manager.go:414] "containerMap: removing container" podUID="394b8516-d37f-45dd-8a79-db1b2e9a3af7" containerName="console" Oct 31 01:21:25.753159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753160 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394b8516-d37f-45dd-8a79-db1b2e9a3af7" containerName="console" Oct 31 01:21:25.753469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753235 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd51c16f-f327-4939-ad47-4acc6aaf62c2" containerName="registry" Oct 31 01:21:25.753469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753246 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="394b8516-d37f-45dd-8a79-db1b2e9a3af7" containerName="console" Oct 31 01:21:25.753469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753258 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e634f3d-0490-480a-9484-a1ee415c5fb1" containerName="insights-gathering" Oct 31 01:21:25.753469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.753266 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="b123e4c4-57cd-49f8-985b-158b4deab82d" containerName="registry" Oct 31 01:21:25.774964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.774931 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x9h8v-jmvqw"] Oct 31 01:21:25.775106 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.775051 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.811476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.811449 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9c33dc6-08f7-475a-bdc2-81221ca26721-archives-path\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.811603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.811489 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2bx\" (UniqueName: \"kubernetes.io/projected/c9c33dc6-08f7-475a-bdc2-81221ca26721-kube-api-access-dv2bx\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.811603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.811581 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-service-ca-bundle\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.811603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.811600 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-trusted-ca-bundle\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.913049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.913017 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9c33dc6-08f7-475a-bdc2-81221ca26721-archives-path\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.913049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.913053 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2bx\" (UniqueName: \"kubernetes.io/projected/c9c33dc6-08f7-475a-bdc2-81221ca26721-kube-api-access-dv2bx\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.913304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.913082 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-service-ca-bundle\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.913304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.913097 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-trusted-ca-bundle\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.913493 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.913471 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9c33dc6-08f7-475a-bdc2-81221ca26721-archives-path\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.913928 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.913907 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-service-ca-bundle\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.914069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.914054 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-trusted-ca-bundle\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:25.925358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:25.925337 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2bx\" (UniqueName: \"kubernetes.io/projected/c9c33dc6-08f7-475a-bdc2-81221ca26721-kube-api-access-dv2bx\") pod \"periodic-gathering-x9h8v-jmvqw\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:26.101848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:26.101788 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:21:26.281243 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:26.281214 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2s452" event={"ID":"f35a7382-ce08-44b1-8427-33a5d30252b7","Type":"ContainerStarted","Data":"297894e9d8c61a2d80dc1b5f4ebf5ca00ab31775e65989c661b367c1455ecb63"} Oct 31 01:21:26.283070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:26.283048 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7hl7l" event={"ID":"8280d35c-b526-41e0-b4a2-0bd74b181789","Type":"ContainerStarted","Data":"34d8e13ad42b3e70751fb9867ed9a535b01af559da58a25021484f01e330e357"} Oct 31 01:21:26.387796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:26.387706 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-x9h8v-jmvqw"] Oct 31 01:21:26.392247 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:21:26.392223 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c33dc6_08f7_475a_bdc2_81221ca26721.slice/crio-8b1321b7e71c11274975f128a43ca10be7f3754b06b77b3ece44a5bcc02f131c WatchSource:0}: Error finding container 8b1321b7e71c11274975f128a43ca10be7f3754b06b77b3ece44a5bcc02f131c: Status 404 returned error can't find the container with id 8b1321b7e71c11274975f128a43ca10be7f3754b06b77b3ece44a5bcc02f131c Oct 31 01:21:27.288572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:27.288535 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" event={"ID":"c9c33dc6-08f7-475a-bdc2-81221ca26721","Type":"ContainerStarted","Data":"0f9f4ef76641bfd56cd3931877f782cc781e9312c46ec35563f330ee8a439137"} Oct 31 01:21:27.289143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:27.288583 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" event={"ID":"c9c33dc6-08f7-475a-bdc2-81221ca26721","Type":"ContainerStarted","Data":"8b1321b7e71c11274975f128a43ca10be7f3754b06b77b3ece44a5bcc02f131c"} Oct 31 01:21:27.307038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:27.306989 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" podStartSLOduration=2.306975634 podStartE2EDuration="2.306975634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:27.305461999 +0000 UTC m=+87.685707222" watchObservedRunningTime="2026-10-31 01:21:27.306975634 +0000 UTC m=+87.687220851" Oct 31 01:21:34.786928 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:21:34.786891 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:21:48.571751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:48.571703 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:54.136442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:54.136400 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:21:54.278520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:21:54.278494 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:03.599446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.599408 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c5b75d49b-bl4g7"] Oct 31 01:22:03.603797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.603763 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.613681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.613653 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c5b75d49b-bl4g7"] Oct 31 01:22:03.700682 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.700640 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-console-config\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.700896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.700744 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-trusted-ca-bundle\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.700896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.700774 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-oauth-config\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.700896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.700798 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d01752c-273a-408d-9867-72d8894c97a3-tmp\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.701067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.700947 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-service-ca\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.701067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.701010 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-serving-cert\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.701067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.701035 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/2d01752c-273a-408d-9867-72d8894c97a3-kube-api-access-9brj9\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.701067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.701061 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-oauth-serving-cert\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.725062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.725038 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:03.802400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802363 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-trusted-ca-bundle\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.802400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802396 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-oauth-config\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.802623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802427 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d01752c-273a-408d-9867-72d8894c97a3-tmp\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.802623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802472 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-service-ca\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.802623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802516 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-serving-cert\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.802623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802533 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/2d01752c-273a-408d-9867-72d8894c97a3-kube-api-access-9brj9\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.802623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802558 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-oauth-serving-cert\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.802623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.802592 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-console-config\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.803133 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.803108 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d01752c-273a-408d-9867-72d8894c97a3-tmp\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.803412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.803356 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-oauth-serving-cert\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.803622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.803595 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-service-ca\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.803622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.803614 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-console-config\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.803810 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.803792 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-trusted-ca-bundle\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.805339 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.805307 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-serving-cert\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.805416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.805316 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-oauth-config\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.814691 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.814670 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/2d01752c-273a-408d-9867-72d8894c97a3-kube-api-access-9brj9\") pod \"console-6c5b75d49b-bl4g7\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:03.917572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:03.917492 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:04.192578 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:04.192469 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c5b75d49b-bl4g7"] Oct 31 01:22:04.195276 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:22:04.195251 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d01752c_273a_408d_9867_72d8894c97a3.slice/crio-6469b1d23bc3e525c19da284997f1a67b040cb928d582fcb959ae87b3613d9cb WatchSource:0}: Error finding container 6469b1d23bc3e525c19da284997f1a67b040cb928d582fcb959ae87b3613d9cb: Status 404 returned error can't find the container with id 6469b1d23bc3e525c19da284997f1a67b040cb928d582fcb959ae87b3613d9cb Oct 31 01:22:04.333463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:04.333428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:04.456514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:04.456436 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c5b75d49b-bl4g7" event={"ID":"2d01752c-273a-408d-9867-72d8894c97a3","Type":"ContainerStarted","Data":"c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311"} Oct 31 01:22:04.456514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:04.456492 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c5b75d49b-bl4g7" event={"ID":"2d01752c-273a-408d-9867-72d8894c97a3","Type":"ContainerStarted","Data":"6469b1d23bc3e525c19da284997f1a67b040cb928d582fcb959ae87b3613d9cb"} Oct 31 01:22:04.479523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:04.479478 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c5b75d49b-bl4g7" podStartSLOduration=1.479465956 podStartE2EDuration="1.479465956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:04.477835761 +0000 UTC m=+124.858080983" watchObservedRunningTime="2026-10-31 01:22:04.479465956 +0000 UTC m=+124.859711176" Oct 31 01:22:04.787326 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:22:04.787291 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:22:08.638887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:08.638849 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:09.345170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:09.345137 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:09.477553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:09.477520 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:12.849810 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.849734 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c5b75d49b-bl4g7"] Oct 31 01:22:12.880708 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.880679 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79b8b67c9-w9c7m"] Oct 31 01:22:12.882991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.882976 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.895507 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.895483 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b8b67c9-w9c7m"] Oct 31 01:22:12.959138 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.959113 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:12.984347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984322 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-service-ca\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.984454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984348 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-tmp\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.984454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984365 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-oauth-serving-cert\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.984454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984436 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85tk4\" (UniqueName: \"kubernetes.io/projected/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-kube-api-access-85tk4\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.984557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984481 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-serving-cert\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.984557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984512 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-trusted-ca-bundle\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.984557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984550 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-config\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:12.984644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:12.984615 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-oauth-config\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085303 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-oauth-config\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085339 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-service-ca\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085355 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-tmp\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085371 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-oauth-serving-cert\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085400 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85tk4\" (UniqueName: \"kubernetes.io/projected/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-kube-api-access-85tk4\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085424 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-serving-cert\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085468 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-trusted-ca-bundle\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.085542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085524 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-config\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.086008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.085981 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-tmp\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.086276 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.086200 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-service-ca\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.086351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.086270 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-config\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.086495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.086477 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-trusted-ca-bundle\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.086556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.086533 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-oauth-serving-cert\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.088149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.088127 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-oauth-config\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.088239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.088188 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-console-serving-cert\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.097242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.097218 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85tk4\" (UniqueName: \"kubernetes.io/projected/3a1d944d-3d2c-4dc0-87d2-1c29fd90927f-kube-api-access-85tk4\") pod \"console-79b8b67c9-w9c7m\" (UID: \"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f\") " pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.100120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.100044 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:13.193822 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.193799 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:13.332293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.332262 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:13.477155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.477085 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b8b67c9-w9c7m"] Oct 31 01:22:13.480078 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:22:13.480051 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1d944d_3d2c_4dc0_87d2_1c29fd90927f.slice/crio-10457f4a24ee65290368a93fdaedcd66b821f2d5573d76372bf7678b08cc56b9 WatchSource:0}: Error finding container 10457f4a24ee65290368a93fdaedcd66b821f2d5573d76372bf7678b08cc56b9: Status 404 returned error can't find the container with id 10457f4a24ee65290368a93fdaedcd66b821f2d5573d76372bf7678b08cc56b9 Oct 31 01:22:13.480977 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.480932 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:13.493407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.493388 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b8b67c9-w9c7m" event={"ID":"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f","Type":"ContainerStarted","Data":"10457f4a24ee65290368a93fdaedcd66b821f2d5573d76372bf7678b08cc56b9"} Oct 31 01:22:13.918308 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:13.918279 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:14.499259 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:14.499227 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b8b67c9-w9c7m" event={"ID":"3a1d944d-3d2c-4dc0-87d2-1c29fd90927f","Type":"ContainerStarted","Data":"00b3f004d37f65b674e1d26947843b3b0637754cf72eb8ee8c74a5bc0d076edc"} Oct 31 01:22:14.519192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:14.519150 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79b8b67c9-w9c7m" podStartSLOduration=2.519137592 podStartE2EDuration="2.519137592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:14.518519443 +0000 UTC m=+134.898764685" watchObservedRunningTime="2026-10-31 01:22:14.519137592 +0000 UTC m=+134.899382815" Oct 31 01:22:19.383779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:19.383749 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:21.951673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:21.951641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:22.142832 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:22.142794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:23.194003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:23.193972 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:23.194403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:23.194060 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:23.198788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:23.198767 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:23.539768 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:23.539746 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79b8b67c9-w9c7m" Oct 31 01:22:23.590743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:23.590697 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9ddc9998-pqcjg"] Oct 31 01:22:34.488031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:34.487997 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:34.633792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:34.633767 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:34.787389 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:22:34.787360 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:22:37.870130 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:37.870093 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c5b75d49b-bl4g7" podUID="2d01752c-273a-408d-9867-72d8894c97a3" containerName="console" containerID="cri-o://c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311" gracePeriod=15 Oct 31 01:22:38.116581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.116555 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:38.177870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.177795 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-console-config\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-console-config\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.177870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.177829 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-trusted-ca-bundle\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.177870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.177851 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-oauth-config\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.177870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.177871 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-service-ca\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.178174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.177906 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d01752c-273a-408d-9867-72d8894c97a3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d01752c-273a-408d-9867-72d8894c97a3-tmp\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.178174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.177937 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-oauth-serving-cert\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.178174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.177959 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d01752c-273a-408d-9867-72d8894c97a3-kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/2d01752c-273a-408d-9867-72d8894c97a3-kube-api-access-9brj9\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.178174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.178062 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-serving-cert\") pod \"2d01752c-273a-408d-9867-72d8894c97a3\" (UID: \"2d01752c-273a-408d-9867-72d8894c97a3\") " Oct 31 01:22:38.178374 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.178323 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-console-config" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:38.178374 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.178340 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d01752c-273a-408d-9867-72d8894c97a3-tmp" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:38.178486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.178448 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-service-ca" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:38.178486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.178463 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-oauth-serving-cert" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:38.178554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.178483 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-trusted-ca-bundle" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:38.180334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.180311 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-oauth-config" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:38.180523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.180494 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-serving-cert" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:38.180611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.180550 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d01752c-273a-408d-9867-72d8894c97a3-kube-api-access-9brj9" pod "2d01752c-273a-408d-9867-72d8894c97a3" (UID: "2d01752c-273a-408d-9867-72d8894c97a3"). InnerVolumeSpecName "kube-api-access-9brj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:38.279156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279131 5090 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-serving-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.279156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279153 5090 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-console-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.279282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279164 5090 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.279282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279172 5090 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2d01752c-273a-408d-9867-72d8894c97a3-console-oauth-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.279282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279183 5090 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.279282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279192 5090 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2d01752c-273a-408d-9867-72d8894c97a3-tmp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.279282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279200 5090 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2d01752c-273a-408d-9867-72d8894c97a3-oauth-serving-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.279282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.279209 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brj9\" (UniqueName: \"kubernetes.io/projected/2d01752c-273a-408d-9867-72d8894c97a3-kube-api-access-9brj9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:38.599185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.599155 5090 generic.go:356] "Generic (PLEG): container finished" podID="2d01752c-273a-408d-9867-72d8894c97a3" containerID="c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311" exitCode=0 Oct 31 01:22:38.599358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.599203 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c5b75d49b-bl4g7" Oct 31 01:22:38.599358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.599241 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c5b75d49b-bl4g7" event={"ID":"2d01752c-273a-408d-9867-72d8894c97a3","Type":"ContainerDied","Data":"c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311"} Oct 31 01:22:38.599358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.599295 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c5b75d49b-bl4g7" event={"ID":"2d01752c-273a-408d-9867-72d8894c97a3","Type":"ContainerDied","Data":"6469b1d23bc3e525c19da284997f1a67b040cb928d582fcb959ae87b3613d9cb"} Oct 31 01:22:38.599358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.599316 5090 scope.go:122] "RemoveContainer" containerID="c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311" Oct 31 01:22:38.608891 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.608870 5090 scope.go:122] "RemoveContainer" containerID="c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311" Oct 31 01:22:38.609160 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:22:38.609143 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311\": container with ID starting with c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311 not found: ID does not exist" containerID="c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311" Oct 31 01:22:38.609229 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.609174 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311"} err="failed to get container status \"c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311\": rpc error: code = NotFound desc = could not find container \"c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311\": container with ID starting with c2660b2fc85d08e612a209d35d22bb61f7db60f01947f3d6bcb4dd36839ac311 not found: ID does not exist" Oct 31 01:22:38.620555 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.620532 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c5b75d49b-bl4g7"] Oct 31 01:22:38.624754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:38.624735 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c5b75d49b-bl4g7"] Oct 31 01:22:39.744964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:39.744935 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d01752c-273a-408d-9867-72d8894c97a3" path="/var/lib/kubelet/pods/2d01752c-273a-408d-9867-72d8894c97a3/volumes" Oct 31 01:22:48.617079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.617022 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f9ddc9998-pqcjg" podUID="a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" containerName="console" containerID="cri-o://b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f" gracePeriod=15 Oct 31 01:22:48.624147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.624103 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:22:48.861979 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.861956 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:22:48.966220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966150 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-config\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-config\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966182 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-kube-api-access-rbg46\" (UniqueName: \"kubernetes.io/projected/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-kube-api-access-rbg46\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966203 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-trusted-ca-bundle\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966224 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-oauth-serving-cert\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966255 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-tmp\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966312 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-oauth-config\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966374 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-serving-cert\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966408 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-service-ca\") pod \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\" (UID: \"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e\") " Oct 31 01:22:48.966709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966624 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-tmp" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:48.966709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966681 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-oauth-serving-cert" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:48.966709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966694 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-config" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:48.966893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966701 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-trusted-ca-bundle" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:48.966930 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.966919 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-service-ca" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:48.968472 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.968450 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-kube-api-access-rbg46" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "kube-api-access-rbg46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:48.968730 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.968693 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-oauth-config" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:48.968994 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:48.968977 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-serving-cert" pod "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" (UID: "a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:22:49.067257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067230 5090 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.067257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067256 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbg46\" (UniqueName: \"kubernetes.io/projected/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-kube-api-access-rbg46\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.067411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067267 5090 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.067411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067276 5090 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-oauth-serving-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.067411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067286 5090 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-tmp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.067411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067295 5090 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-oauth-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.067411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067303 5090 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-console-serving-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.067411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.067312 5090 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:49.646603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.646571 5090 generic.go:356] "Generic (PLEG): container finished" podID="c9c33dc6-08f7-475a-bdc2-81221ca26721" containerID="0f9f4ef76641bfd56cd3931877f782cc781e9312c46ec35563f330ee8a439137" exitCode=0 Oct 31 01:22:49.646984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.646645 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" event={"ID":"c9c33dc6-08f7-475a-bdc2-81221ca26721","Type":"ContainerDied","Data":"0f9f4ef76641bfd56cd3931877f782cc781e9312c46ec35563f330ee8a439137"} Oct 31 01:22:49.647890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.647867 5090 generic.go:356] "Generic (PLEG): container finished" podID="a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" containerID="b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f" exitCode=0 Oct 31 01:22:49.647962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.647914 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f9ddc9998-pqcjg" Oct 31 01:22:49.647962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.647944 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9ddc9998-pqcjg" event={"ID":"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e","Type":"ContainerDied","Data":"b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f"} Oct 31 01:22:49.648028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.647991 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f9ddc9998-pqcjg" event={"ID":"a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e","Type":"ContainerDied","Data":"eacef9fbe43f6c0ff33667d101ede15b8fb7e131f676e282a36ae8cc6ec630e8"} Oct 31 01:22:49.648028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.648011 5090 scope.go:122] "RemoveContainer" containerID="b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f" Oct 31 01:22:49.657582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.657568 5090 scope.go:122] "RemoveContainer" containerID="b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f" Oct 31 01:22:49.657860 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:22:49.657842 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f\": container with ID starting with b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f not found: ID does not exist" containerID="b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f" Oct 31 01:22:49.657937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.657868 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f"} err="failed to get container status \"b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f\": rpc error: code = NotFound desc = could not find container \"b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f\": container with ID starting with b2ee2e05cde488471c7e2d43ca691d85c8868b55869a69572661d2f33be48e5f not found: ID does not exist" Oct 31 01:22:49.680604 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.680579 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f9ddc9998-pqcjg"] Oct 31 01:22:49.684843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.684823 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f9ddc9998-pqcjg"] Oct 31 01:22:49.749285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:49.749260 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" path="/var/lib/kubelet/pods/a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e/volumes" Oct 31 01:22:50.783302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.783271 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:22:50.881406 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.881378 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c33dc6-08f7-475a-bdc2-81221ca26721-kube-api-access-dv2bx\" (UniqueName: \"kubernetes.io/projected/c9c33dc6-08f7-475a-bdc2-81221ca26721-kube-api-access-dv2bx\") pod \"c9c33dc6-08f7-475a-bdc2-81221ca26721\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " Oct 31 01:22:50.881558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.881410 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-service-ca-bundle\") pod \"c9c33dc6-08f7-475a-bdc2-81221ca26721\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " Oct 31 01:22:50.881558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.881434 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9c33dc6-08f7-475a-bdc2-81221ca26721-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9c33dc6-08f7-475a-bdc2-81221ca26721-archives-path\") pod \"c9c33dc6-08f7-475a-bdc2-81221ca26721\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " Oct 31 01:22:50.881650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.881556 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-trusted-ca-bundle\") pod \"c9c33dc6-08f7-475a-bdc2-81221ca26721\" (UID: \"c9c33dc6-08f7-475a-bdc2-81221ca26721\") " Oct 31 01:22:50.882085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.881852 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c33dc6-08f7-475a-bdc2-81221ca26721-archives-path" pod "c9c33dc6-08f7-475a-bdc2-81221ca26721" (UID: "c9c33dc6-08f7-475a-bdc2-81221ca26721"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:22:50.882085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.881868 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-service-ca-bundle" pod "c9c33dc6-08f7-475a-bdc2-81221ca26721" (UID: "c9c33dc6-08f7-475a-bdc2-81221ca26721"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:50.882085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.881957 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-trusted-ca-bundle" pod "c9c33dc6-08f7-475a-bdc2-81221ca26721" (UID: "c9c33dc6-08f7-475a-bdc2-81221ca26721"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:22:50.883634 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.883618 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c33dc6-08f7-475a-bdc2-81221ca26721-kube-api-access-dv2bx" pod "c9c33dc6-08f7-475a-bdc2-81221ca26721" (UID: "c9c33dc6-08f7-475a-bdc2-81221ca26721"). InnerVolumeSpecName "kube-api-access-dv2bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:22:50.982932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.982876 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2bx\" (UniqueName: \"kubernetes.io/projected/c9c33dc6-08f7-475a-bdc2-81221ca26721-kube-api-access-dv2bx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:50.982932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.982898 5090 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-service-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:50.982932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.982908 5090 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9c33dc6-08f7-475a-bdc2-81221ca26721-archives-path\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:50.982932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:50.982917 5090 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9c33dc6-08f7-475a-bdc2-81221ca26721-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:22:51.658456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:51.658378 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" event={"ID":"c9c33dc6-08f7-475a-bdc2-81221ca26721","Type":"ContainerDied","Data":"8b1321b7e71c11274975f128a43ca10be7f3754b06b77b3ece44a5bcc02f131c"} Oct 31 01:22:51.658456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:51.658402 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-x9h8v-jmvqw" Oct 31 01:22:51.658456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:22:51.658421 5090 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b1321b7e71c11274975f128a43ca10be7f3754b06b77b3ece44a5bcc02f131c" Oct 31 01:23:04.786348 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:23:04.786300 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:23:09.505891 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:23:09.505853 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:25.864654 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:23:25.864622 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:26.012770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:23:26.012736 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:23:34.787132 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:23:34.787099 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:23:48.665348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:23:48.665242 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:04.786282 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:24:04.786240 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:24:06.091838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:06.091785 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:06.255825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:06.255795 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:07.702324 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:07.702288 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:07.858054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:07.857985 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:09.105442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:09.105407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:12.454971 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:12.454937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:12.603291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:12.603255 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:13.475429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475394 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77"] Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475754 5090 cpu_manager.go:414] "containerMap: removing container" podUID="2d01752c-273a-408d-9867-72d8894c97a3" containerName="console" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475766 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d01752c-273a-408d-9867-72d8894c97a3" containerName="console" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475778 5090 cpu_manager.go:414] "containerMap: removing container" podUID="a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" containerName="console" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475783 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" containerName="console" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475792 5090 cpu_manager.go:414] "containerMap: removing container" podUID="c9c33dc6-08f7-475a-bdc2-81221ca26721" containerName="insights-gathering" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475797 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c33dc6-08f7-475a-bdc2-81221ca26721" containerName="insights-gathering" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475855 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d01752c-273a-408d-9867-72d8894c97a3" containerName="console" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475863 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ad36ca-1f47-4a1f-8a95-8b71ebbf4f0e" containerName="console" Oct 31 01:24:13.475949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.475870 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c33dc6-08f7-475a-bdc2-81221ca26721" containerName="insights-gathering" Oct 31 01:24:13.478886 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.478870 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.482154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.482128 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 01:24:13.482154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.482129 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 01:24:13.482828 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.482811 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-lpj9w\"" Oct 31 01:24:13.493672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.493652 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77"] Oct 31 01:24:13.587335 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.587308 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88s9h\" (UniqueName: \"kubernetes.io/projected/38184af8-4232-4c20-9f5c-de1e0b998d7c-kube-api-access-88s9h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.587484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.587344 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.587484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.587375 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.668783 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.668747 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll"] Oct 31 01:24:13.672155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.672137 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.683565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.683546 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-mqqdw\"" Oct 31 01:24:13.687872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.687850 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88s9h\" (UniqueName: \"kubernetes.io/projected/38184af8-4232-4c20-9f5c-de1e0b998d7c-kube-api-access-88s9h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.687941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.687884 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.687941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.687915 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.688197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.688181 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.688252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.688209 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.689551 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.689515 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 01:24:13.691078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.691059 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll"] Oct 31 01:24:13.701629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.701605 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88s9h\" (UniqueName: \"kubernetes.io/projected/38184af8-4232-4c20-9f5c-de1e0b998d7c-kube-api-access-88s9h\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.774014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.773988 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:13.789031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789007 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789065 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789088 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789193 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789234 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789396 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789319 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5sj6\" (UniqueName: \"kubernetes.io/projected/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-kube-api-access-w5sj6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789396 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789374 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789460 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.789541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.789524 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.793983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.793963 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:13.890852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.890821 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.890852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.890854 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891033 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.890878 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5sj6\" (UniqueName: \"kubernetes.io/projected/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-kube-api-access-w5sj6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891033 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.890905 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891033 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.890928 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891033 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.890958 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891033 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.890987 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.891070 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.891095 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.891264 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891704 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.891666 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891704 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.891692 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.891735 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.891961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.891937 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.893505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.893478 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.893605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.893521 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.906190 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.906167 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.906275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.906172 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5sj6\" (UniqueName: \"kubernetes.io/projected/34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24-kube-api-access-w5sj6\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-8qmll\" (UID: \"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:13.923393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.923372 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:13.984061 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:13.984036 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:14.073247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:14.073213 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77"] Oct 31 01:24:14.079300 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:24:14.079268 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38184af8_4232_4c20_9f5c_de1e0b998d7c.slice/crio-70e6f0be27400d62908ef7a09332095e6ff8bb06a0306f618c24026ae2e880aa WatchSource:0}: Error finding container 70e6f0be27400d62908ef7a09332095e6ff8bb06a0306f618c24026ae2e880aa: Status 404 returned error can't find the container with id 70e6f0be27400d62908ef7a09332095e6ff8bb06a0306f618c24026ae2e880aa Oct 31 01:24:14.260866 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:14.260840 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll"] Oct 31 01:24:14.263205 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:24:14.263184 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bf4cf5_fccc_45b2_a1d1_8bc6c9e06a24.slice/crio-322101d80384f19b5c6373074a0ed377a207a68ca3e953cf97ad81b36031459b WatchSource:0}: Error finding container 322101d80384f19b5c6373074a0ed377a207a68ca3e953cf97ad81b36031459b: Status 404 returned error can't find the container with id 322101d80384f19b5c6373074a0ed377a207a68ca3e953cf97ad81b36031459b Oct 31 01:24:14.985346 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:14.985308 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" event={"ID":"38184af8-4232-4c20-9f5c-de1e0b998d7c","Type":"ContainerStarted","Data":"70e6f0be27400d62908ef7a09332095e6ff8bb06a0306f618c24026ae2e880aa"} Oct 31 01:24:14.986797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:14.986760 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" event={"ID":"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24","Type":"ContainerStarted","Data":"322101d80384f19b5c6373074a0ed377a207a68ca3e953cf97ad81b36031459b"} Oct 31 01:24:16.609622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:16.609585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:17.790883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:17.790844 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:24:17.791304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:17.790930 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:24:17.791304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:17.790973 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:24:18.005412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:18.005371 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" event={"ID":"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24","Type":"ContainerStarted","Data":"3e7faebb2602296772d659583a6414ff49b9bcfd3577682df6bd57d4b77cd75e"} Oct 31 01:24:18.029505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:18.029427 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" podStartSLOduration=1.504697718 podStartE2EDuration="5.029413372s" totalImagesPullingTime="3.524715654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:18.028806897 +0000 UTC m=+258.409052144" watchObservedRunningTime="2026-10-31 01:24:18.029413372 +0000 UTC m=+258.409658595" Oct 31 01:24:18.985020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:18.984984 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:18.990511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:18.990485 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:19.011150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:19.011122 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:19.011972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:19.011941 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:24:21.021295 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:21.021261 5090 generic.go:356] "Generic (PLEG): container finished" podID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerID="2d7532339eb20621bccbf598d7d2c079d7138494c6a14f6c63dec7f180a17ddf" exitCode=0 Oct 31 01:24:21.021706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:21.021344 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" event={"ID":"38184af8-4232-4c20-9f5c-de1e0b998d7c","Type":"ContainerDied","Data":"2d7532339eb20621bccbf598d7d2c079d7138494c6a14f6c63dec7f180a17ddf"} Oct 31 01:24:24.039221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:24.039182 5090 generic.go:356] "Generic (PLEG): container finished" podID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerID="4158fe1f80593705d91b74dba42f950a8db1d75995de2aa2c2458c1d00669a00" exitCode=0 Oct 31 01:24:24.039616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:24.039267 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" event={"ID":"38184af8-4232-4c20-9f5c-de1e0b998d7c","Type":"ContainerDied","Data":"4158fe1f80593705d91b74dba42f950a8db1d75995de2aa2c2458c1d00669a00"} Oct 31 01:24:31.072983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:31.072946 5090 generic.go:356] "Generic (PLEG): container finished" podID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerID="7ee5f461ba56b2720f77e35927166b5bde6203e33f7ff99536564d89831c8106" exitCode=0 Oct 31 01:24:31.073468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:31.073002 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" event={"ID":"38184af8-4232-4c20-9f5c-de1e0b998d7c","Type":"ContainerDied","Data":"7ee5f461ba56b2720f77e35927166b5bde6203e33f7ff99536564d89831c8106"} Oct 31 01:24:32.202893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.202871 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:32.252372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.252341 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-bundle\") pod \"38184af8-4232-4c20-9f5c-de1e0b998d7c\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " Oct 31 01:24:32.252538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.252401 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-util\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-util\") pod \"38184af8-4232-4c20-9f5c-de1e0b998d7c\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " Oct 31 01:24:32.252538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.252438 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38184af8-4232-4c20-9f5c-de1e0b998d7c-kube-api-access-88s9h\" (UniqueName: \"kubernetes.io/projected/38184af8-4232-4c20-9f5c-de1e0b998d7c-kube-api-access-88s9h\") pod \"38184af8-4232-4c20-9f5c-de1e0b998d7c\" (UID: \"38184af8-4232-4c20-9f5c-de1e0b998d7c\") " Oct 31 01:24:32.253005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.252981 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-bundle" pod "38184af8-4232-4c20-9f5c-de1e0b998d7c" (UID: "38184af8-4232-4c20-9f5c-de1e0b998d7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:32.254810 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.254784 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38184af8-4232-4c20-9f5c-de1e0b998d7c-kube-api-access-88s9h" pod "38184af8-4232-4c20-9f5c-de1e0b998d7c" (UID: "38184af8-4232-4c20-9f5c-de1e0b998d7c"). InnerVolumeSpecName "kube-api-access-88s9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:24:32.256677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.256652 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-util" pod "38184af8-4232-4c20-9f5c-de1e0b998d7c" (UID: "38184af8-4232-4c20-9f5c-de1e0b998d7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:24:32.353666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.353601 5090 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-util\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:32.353666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.353627 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88s9h\" (UniqueName: \"kubernetes.io/projected/38184af8-4232-4c20-9f5c-de1e0b998d7c-kube-api-access-88s9h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:32.353666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:32.353638 5090 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38184af8-4232-4c20-9f5c-de1e0b998d7c-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:24:33.084495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:33.084468 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" Oct 31 01:24:33.084495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:33.084480 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6sm77" event={"ID":"38184af8-4232-4c20-9f5c-de1e0b998d7c","Type":"ContainerDied","Data":"70e6f0be27400d62908ef7a09332095e6ff8bb06a0306f618c24026ae2e880aa"} Oct 31 01:24:33.084692 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:33.084520 5090 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e6f0be27400d62908ef7a09332095e6ff8bb06a0306f618c24026ae2e880aa" Oct 31 01:24:34.789775 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:24:34.789706 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:24:34.907474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:34.907441 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:36.400748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:36.400702 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:38.862473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:38.862439 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:39.003655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:39.003626 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:41.119890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:41.119859 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:41.336659 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:41.336625 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:41.549160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:41.549127 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:42.025818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.025785 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:42.237412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.237379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:42.300995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.300908 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb"] Oct 31 01:24:42.301656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.301630 5090 cpu_manager.go:414] "containerMap: removing container" podUID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerName="util" Oct 31 01:24:42.301805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.301657 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerName="util" Oct 31 01:24:42.301805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.301683 5090 cpu_manager.go:414] "containerMap: removing container" podUID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerName="extract" Oct 31 01:24:42.301805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.301690 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerName="extract" Oct 31 01:24:42.301805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.301736 5090 cpu_manager.go:414] "containerMap: removing container" podUID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerName="pull" Oct 31 01:24:42.301805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.301745 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerName="pull" Oct 31 01:24:42.302041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.301902 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="38184af8-4232-4c20-9f5c-de1e0b998d7c" containerName="extract" Oct 31 01:24:42.304993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.304971 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.320238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.320208 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nf8\" (UniqueName: \"kubernetes.io/projected/5777eecc-4452-4e4d-a135-95603e8fcf3c-kube-api-access-d8nf8\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.320362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.320247 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5777eecc-4452-4e4d-a135-95603e8fcf3c-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.320362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.320279 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.320444 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.320396 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.320444 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.320428 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5777eecc-4452-4e4d-a135-95603e8fcf3c-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.320513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.320449 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.352442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.352410 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb"] Oct 31 01:24:42.364024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.363987 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 01:24:42.364144 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.364129 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 01:24:42.364475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.364459 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-25fpn\"" Oct 31 01:24:42.364573 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.364559 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 01:24:42.367636 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.367620 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 01:24:42.367953 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.367931 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 01:24:42.421051 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.421018 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.421202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.421053 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5777eecc-4452-4e4d-a135-95603e8fcf3c-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.421202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.421074 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.421202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.421134 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nf8\" (UniqueName: \"kubernetes.io/projected/5777eecc-4452-4e4d-a135-95603e8fcf3c-kube-api-access-d8nf8\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.421202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.421155 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5777eecc-4452-4e4d-a135-95603e8fcf3c-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.421202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.421184 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.421631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.421593 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5777eecc-4452-4e4d-a135-95603e8fcf3c-manifests\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.423734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.423699 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-webhook-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.423815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.423760 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.423985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.423967 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5777eecc-4452-4e4d-a135-95603e8fcf3c-tmp\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.424186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.424171 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5777eecc-4452-4e4d-a135-95603e8fcf3c-metrics-cert\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.442488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.442465 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nf8\" (UniqueName: \"kubernetes.io/projected/5777eecc-4452-4e4d-a135-95603e8fcf3c-kube-api-access-d8nf8\") pod \"opendatahub-operator-controller-manager-69496c6ffb-cp2sb\" (UID: \"5777eecc-4452-4e4d-a135-95603e8fcf3c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.586771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.586669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:42.617536 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.617515 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:42.739063 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.739027 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:42.925308 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:24:42.925280 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5777eecc_4452_4e4d_a135_95603e8fcf3c.slice/crio-612c67959a427d58c0fdd69111f6b98e810d186dced2ad1d9f0a85d5b7eb8177 WatchSource:0}: Error finding container 612c67959a427d58c0fdd69111f6b98e810d186dced2ad1d9f0a85d5b7eb8177: Status 404 returned error can't find the container with id 612c67959a427d58c0fdd69111f6b98e810d186dced2ad1d9f0a85d5b7eb8177 Oct 31 01:24:42.925429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.925318 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb"] Oct 31 01:24:42.930460 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:42.930438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:43.124622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:43.124590 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" event={"ID":"5777eecc-4452-4e4d-a135-95603e8fcf3c","Type":"ContainerStarted","Data":"612c67959a427d58c0fdd69111f6b98e810d186dced2ad1d9f0a85d5b7eb8177"} Oct 31 01:24:43.660896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:43.660843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:44.121519 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:44.121484 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:44.772322 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:44.772271 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:44.979010 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:44.978983 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:45.177429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:45.177349 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:45.756991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:45.756955 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:45.925964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:45.925933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:47.146416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:47.146334 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" event={"ID":"5777eecc-4452-4e4d-a135-95603e8fcf3c","Type":"ContainerStarted","Data":"62dd2923208d9588ea0dba2b7fb083c2d12b8d2746cf3087966d9a0ace75cc35"} Oct 31 01:24:48.152267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:48.152235 5090 generic.go:356] "Generic (PLEG): container finished" podID="5777eecc-4452-4e4d-a135-95603e8fcf3c" containerID="62dd2923208d9588ea0dba2b7fb083c2d12b8d2746cf3087966d9a0ace75cc35" exitCode=0 Oct 31 01:24:48.152640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:48.152334 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" event={"ID":"5777eecc-4452-4e4d-a135-95603e8fcf3c","Type":"ContainerDied","Data":"62dd2923208d9588ea0dba2b7fb083c2d12b8d2746cf3087966d9a0ace75cc35"} Oct 31 01:24:48.780024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:48.779988 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:49.158606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:49.158570 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" event={"ID":"5777eecc-4452-4e4d-a135-95603e8fcf3c","Type":"ContainerStarted","Data":"4f1d33196c333085ed721a9bc90c3222b2248863b9738897843f8b737eae98f7"} Oct 31 01:24:49.159045 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:49.158744 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:24:49.218133 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:49.218089 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" podStartSLOduration=2.515129719 podStartE2EDuration="7.218077037s" totalImagesPullingTime="4.240135843s" totalInitContainerRuntime="462.811475ms" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:49.218015522 +0000 UTC m=+289.598260744" watchObservedRunningTime="2026-10-31 01:24:49.218077037 +0000 UTC m=+289.598322251" Oct 31 01:24:49.484155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:49.484081 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:50.711794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:50.711760 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:52.075191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:52.075157 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:53.088730 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:53.088675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:53.240193 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:53.240159 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:54.976887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:54.976857 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b67f8"] Oct 31 01:24:54.981006 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:54.980986 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" Oct 31 01:24:54.986761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:54.986734 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-888pj\"" Oct 31 01:24:54.986870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:54.986762 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 01:24:54.986870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:54.986796 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 01:24:55.000350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:55.000329 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b67f8"] Oct 31 01:24:55.016949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:55.016919 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2zn\" (UniqueName: \"kubernetes.io/projected/8507cc7a-393d-40c2-bd9a-8755fe6ae0bb-kube-api-access-qb2zn\") pod \"authorino-operator-7b7bb4bc77-b67f8\" (UID: \"8507cc7a-393d-40c2-bd9a-8755fe6ae0bb\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" Oct 31 01:24:55.117692 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:55.117669 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2zn\" (UniqueName: \"kubernetes.io/projected/8507cc7a-393d-40c2-bd9a-8755fe6ae0bb-kube-api-access-qb2zn\") pod \"authorino-operator-7b7bb4bc77-b67f8\" (UID: \"8507cc7a-393d-40c2-bd9a-8755fe6ae0bb\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" Oct 31 01:24:55.132839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:55.132814 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2zn\" (UniqueName: \"kubernetes.io/projected/8507cc7a-393d-40c2-bd9a-8755fe6ae0bb-kube-api-access-qb2zn\") pod \"authorino-operator-7b7bb4bc77-b67f8\" (UID: \"8507cc7a-393d-40c2-bd9a-8755fe6ae0bb\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" Oct 31 01:24:55.292024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:55.292003 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" Oct 31 01:24:55.568566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:55.568498 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-b67f8"] Oct 31 01:24:55.571514 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:24:55.571477 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8507cc7a_393d_40c2_bd9a_8755fe6ae0bb.slice/crio-e22fdd6efa2d49e6853303cec5a14e55f605ab248ecf3d15101b93357dfee093 WatchSource:0}: Error finding container e22fdd6efa2d49e6853303cec5a14e55f605ab248ecf3d15101b93357dfee093: Status 404 returned error can't find the container with id e22fdd6efa2d49e6853303cec5a14e55f605ab248ecf3d15101b93357dfee093 Oct 31 01:24:56.091920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:56.091892 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:56.189062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:56.189034 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" event={"ID":"8507cc7a-393d-40c2-bd9a-8755fe6ae0bb","Type":"ContainerStarted","Data":"e22fdd6efa2d49e6853303cec5a14e55f605ab248ecf3d15101b93357dfee093"} Oct 31 01:24:56.656050 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:56.656014 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:56.899732 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:56.899669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:58.006309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.006266 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw"] Oct 31 01:24:58.009943 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.009917 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" Oct 31 01:24:58.015481 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.015443 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5x4t5\"" Oct 31 01:24:58.017184 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.017163 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw"] Oct 31 01:24:58.019099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.019079 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 01:24:58.042778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.042749 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tg4\" (UniqueName: \"kubernetes.io/projected/ec0e0324-e8d7-480a-a599-99a0ef19f3b3-kube-api-access-h9tg4\") pod \"dns-operator-controller-manager-8557cbdb84-bzvnw\" (UID: \"ec0e0324-e8d7-480a-a599-99a0ef19f3b3\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" Oct 31 01:24:58.143700 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.143662 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tg4\" (UniqueName: \"kubernetes.io/projected/ec0e0324-e8d7-480a-a599-99a0ef19f3b3-kube-api-access-h9tg4\") pod \"dns-operator-controller-manager-8557cbdb84-bzvnw\" (UID: \"ec0e0324-e8d7-480a-a599-99a0ef19f3b3\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" Oct 31 01:24:58.158029 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.157999 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tg4\" (UniqueName: \"kubernetes.io/projected/ec0e0324-e8d7-480a-a599-99a0ef19f3b3-kube-api-access-h9tg4\") pod \"dns-operator-controller-manager-8557cbdb84-bzvnw\" (UID: \"ec0e0324-e8d7-480a-a599-99a0ef19f3b3\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" Oct 31 01:24:58.324631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.324555 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" Oct 31 01:24:58.607639 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:58.607563 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw"] Oct 31 01:24:58.612668 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:24:58.612635 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0e0324_e8d7_480a_a599_99a0ef19f3b3.slice/crio-ea0ae2f26ed85a84881bac987d38b94be51c1b1314f45504bf0b784549458fa8 WatchSource:0}: Error finding container ea0ae2f26ed85a84881bac987d38b94be51c1b1314f45504bf0b784549458fa8: Status 404 returned error can't find the container with id ea0ae2f26ed85a84881bac987d38b94be51c1b1314f45504bf0b784549458fa8 Oct 31 01:24:59.065461 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.065435 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:59.204885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.204847 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" event={"ID":"8507cc7a-393d-40c2-bd9a-8755fe6ae0bb","Type":"ContainerStarted","Data":"da867cf65d3b034b339356221cb29e0d678ce0d05abb95fd97d06c53c40dc90d"} Oct 31 01:24:59.205062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.204914 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" Oct 31 01:24:59.206296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.206269 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" event={"ID":"ec0e0324-e8d7-480a-a599-99a0ef19f3b3","Type":"ContainerStarted","Data":"ea0ae2f26ed85a84881bac987d38b94be51c1b1314f45504bf0b784549458fa8"} Oct 31 01:24:59.231530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.231491 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" podStartSLOduration=1.76571799 podStartE2EDuration="5.231478775s" totalImagesPullingTime="3.465760785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:24:59.23039909 +0000 UTC m=+299.610644324" watchObservedRunningTime="2026-10-31 01:24:59.231478775 +0000 UTC m=+299.611724071" Oct 31 01:24:59.321079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.321007 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:59.483927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.483897 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:24:59.772792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.772760 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:24:59.773680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.773657 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:24:59.778490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.778475 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:24:59.779182 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:24:59.779166 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:25:00.164871 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:00.164802 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-69496c6ffb-cp2sb" Oct 31 01:25:01.338381 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.338335 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:01.397195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.397168 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh"] Oct 31 01:25:01.400246 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.400227 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" Oct 31 01:25:01.411273 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.411246 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-fj2nh\"" Oct 31 01:25:01.411991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.411968 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh"] Oct 31 01:25:01.471482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.471458 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8cc7\" (UniqueName: \"kubernetes.io/projected/d7d1860c-077f-444b-bf09-7040910e43c4-kube-api-access-t8cc7\") pod \"limitador-operator-controller-manager-5778bb4449-276kh\" (UID: \"d7d1860c-077f-444b-bf09-7040910e43c4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" Oct 31 01:25:01.572575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.572550 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8cc7\" (UniqueName: \"kubernetes.io/projected/d7d1860c-077f-444b-bf09-7040910e43c4-kube-api-access-t8cc7\") pod \"limitador-operator-controller-manager-5778bb4449-276kh\" (UID: \"d7d1860c-077f-444b-bf09-7040910e43c4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" Oct 31 01:25:01.593059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.592987 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8cc7\" (UniqueName: \"kubernetes.io/projected/d7d1860c-077f-444b-bf09-7040910e43c4-kube-api-access-t8cc7\") pod \"limitador-operator-controller-manager-5778bb4449-276kh\" (UID: \"d7d1860c-077f-444b-bf09-7040910e43c4\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" Oct 31 01:25:01.711076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.711041 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" Oct 31 01:25:01.988551 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.988524 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh"] Oct 31 01:25:01.991259 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:25:01.991234 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d1860c_077f_444b_bf09_7040910e43c4.slice/crio-db9de929e6d813b3783296c05b93aa6470f322af3a8d12e28cb5dbfdf5fa2f12 WatchSource:0}: Error finding container db9de929e6d813b3783296c05b93aa6470f322af3a8d12e28cb5dbfdf5fa2f12: Status 404 returned error can't find the container with id db9de929e6d813b3783296c05b93aa6470f322af3a8d12e28cb5dbfdf5fa2f12 Oct 31 01:25:01.993617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:01.993600 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:25:02.138952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:02.138912 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:02.222662 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:02.222625 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" event={"ID":"d7d1860c-077f-444b-bf09-7040910e43c4","Type":"ContainerStarted","Data":"db9de929e6d813b3783296c05b93aa6470f322af3a8d12e28cb5dbfdf5fa2f12"} Oct 31 01:25:02.474279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:02.474257 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 01:25:02.662547 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:02.662520 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:03.228784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:03.228750 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" event={"ID":"ec0e0324-e8d7-480a-a599-99a0ef19f3b3","Type":"ContainerStarted","Data":"acfebd64ac925fb3c0a696f63db00207464887ce8044170029e2876b7f142365"} Oct 31 01:25:03.228945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:03.228831 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" Oct 31 01:25:03.283100 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:03.283064 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:03.551545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:03.551510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:04.234788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:04.234687 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" event={"ID":"d7d1860c-077f-444b-bf09-7040910e43c4","Type":"ContainerStarted","Data":"d3f404518cd54d2e2cfc5ccb5d0650e88c490941abcdce112c9d1e5cf4997aca"} Oct 31 01:25:04.234788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:04.234782 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" Oct 31 01:25:04.275089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:04.270948 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" podStartSLOduration=1.401455249 podStartE2EDuration="3.270930351s" totalImagesPullingTime="1.869475102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:04.268275195 +0000 UTC m=+304.648520429" watchObservedRunningTime="2026-10-31 01:25:04.270930351 +0000 UTC m=+304.651175575" Oct 31 01:25:04.275089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:04.271146 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" podStartSLOduration=3.415004739 podStartE2EDuration="7.271140786s" totalImagesPullingTime="3.856136047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:03.28324071 +0000 UTC m=+303.663485948" watchObservedRunningTime="2026-10-31 01:25:04.271140786 +0000 UTC m=+304.651386012" Oct 31 01:25:04.440001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:04.439969 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:04.599027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:04.598987 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:04.787278 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:25:04.787235 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:25:08.897210 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:08.897181 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:09.043997 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:09.043959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:09.195469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:09.195397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:10.213795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:10.213700 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-b67f8" Oct 31 01:25:12.739216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:12.739178 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:12.896198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:12.896163 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:13.190822 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.190793 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:13.353393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.353355 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:13.548580 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.548536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:13.686449 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.686423 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s"] Oct 31 01:25:13.689951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.689932 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.704879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.704854 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s"] Oct 31 01:25:13.706687 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.706671 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b6kpp\"" Oct 31 01:25:13.744855 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.744826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:13.775220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775196 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775230 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775250 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775382 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775433 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775469 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563c1a67-983f-49b3-b52e-a57c8e515ce6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775552 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775535 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln58g\" (UniqueName: \"kubernetes.io/projected/563c1a67-983f-49b3-b52e-a57c8e515ce6-kube-api-access-ln58g\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775591 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.775697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.775674 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.876928 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.876961 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563c1a67-983f-49b3-b52e-a57c8e515ce6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.876983 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln58g\" (UniqueName: \"kubernetes.io/projected/563c1a67-983f-49b3-b52e-a57c8e515ce6-kube-api-access-ln58g\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877023 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877045 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877065 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877084 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877106 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877142 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877508 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877483 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877590 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877556 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877639 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877775 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877758 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/563c1a67-983f-49b3-b52e-a57c8e515ce6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.877839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.877820 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.879678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.879653 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.879796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.879761 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.889633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.889612 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln58g\" (UniqueName: \"kubernetes.io/projected/563c1a67-983f-49b3-b52e-a57c8e515ce6-kube-api-access-ln58g\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.889706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.889674 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/563c1a67-983f-49b3-b52e-a57c8e515ce6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-98f7774f-f966s\" (UID: \"563c1a67-983f-49b3-b52e-a57c8e515ce6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:13.906079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:13.906040 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:14.003205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:14.003173 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:14.100466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:14.100432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:14.237482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:14.237411 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-bzvnw" Oct 31 01:25:14.316746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:14.316695 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s"] Oct 31 01:25:14.320121 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:25:14.320094 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563c1a67_983f_49b3_b52e_a57c8e515ce6.slice/crio-c2f583cab771e562bd4ace5821b5fe5e3c7ecc723982e3206b7dc0ec8e04af31 WatchSource:0}: Error finding container c2f583cab771e562bd4ace5821b5fe5e3c7ecc723982e3206b7dc0ec8e04af31: Status 404 returned error can't find the container with id c2f583cab771e562bd4ace5821b5fe5e3c7ecc723982e3206b7dc0ec8e04af31 Oct 31 01:25:14.322291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:14.322261 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:25:14.322356 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:14.322324 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:25:14.322388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:14.322354 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:25:15.243139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:15.243110 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-276kh" Oct 31 01:25:15.288403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:15.288358 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" event={"ID":"563c1a67-983f-49b3-b52e-a57c8e515ce6","Type":"ContainerStarted","Data":"bdb44a4cf4657d7790b949fa1498b2f27f7a93668d791f36f0c8eb2fd1a54051"} Oct 31 01:25:15.288582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:15.288421 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" event={"ID":"563c1a67-983f-49b3-b52e-a57c8e515ce6","Type":"ContainerStarted","Data":"c2f583cab771e562bd4ace5821b5fe5e3c7ecc723982e3206b7dc0ec8e04af31"} Oct 31 01:25:15.316734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:15.316667 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" podStartSLOduration=2.3166494699999998 podStartE2EDuration="2.31664947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:15.314748341 +0000 UTC m=+315.694993581" watchObservedRunningTime="2026-10-31 01:25:15.31664947 +0000 UTC m=+315.696894693" Oct 31 01:25:15.467828 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:15.467789 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:15.674591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:15.674504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:16.003742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:16.003692 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:16.008677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:16.008658 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:16.296829 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:16.296737 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:16.298331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:16.298306 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-98f7774f-f966s" Oct 31 01:25:16.711599 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:16.711518 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:16.888245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:16.888213 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:17.421194 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.421158 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs"] Oct 31 01:25:17.424804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.424784 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.431863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.431493 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gdjrb\"" Oct 31 01:25:17.434041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.434022 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 01:25:17.434443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.434427 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 01:25:17.436416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.436400 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 01:25:17.441932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.441908 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs"] Oct 31 01:25:17.506622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.506594 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bfaf10a1-76a7-49ee-bd87-671bab789262-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.506811 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.506641 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf10a1-76a7-49ee-bd87-671bab789262-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.506811 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.506674 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs6ls\" (UniqueName: \"kubernetes.io/projected/bfaf10a1-76a7-49ee-bd87-671bab789262-kube-api-access-hs6ls\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.506889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.506840 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfaf10a1-76a7-49ee-bd87-671bab789262-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.607966 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.607936 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfaf10a1-76a7-49ee-bd87-671bab789262-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.608118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.607975 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bfaf10a1-76a7-49ee-bd87-671bab789262-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.608118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.608032 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf10a1-76a7-49ee-bd87-671bab789262-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.608118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.608074 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs6ls\" (UniqueName: \"kubernetes.io/projected/bfaf10a1-76a7-49ee-bd87-671bab789262-kube-api-access-hs6ls\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.608428 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.608408 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfaf10a1-76a7-49ee-bd87-671bab789262-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.608494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.608428 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bfaf10a1-76a7-49ee-bd87-671bab789262-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.608904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.608886 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfaf10a1-76a7-49ee-bd87-671bab789262-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.637915 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.637888 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs6ls\" (UniqueName: \"kubernetes.io/projected/bfaf10a1-76a7-49ee-bd87-671bab789262-kube-api-access-hs6ls\") pod \"kserve-module-controller-manager-7566cf5fc8-8whxs\" (UID: \"bfaf10a1-76a7-49ee-bd87-671bab789262\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:17.738876 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:17.738795 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:18.019185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:18.019147 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs"] Oct 31 01:25:18.021631 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:25:18.021597 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaf10a1_76a7_49ee_bd87_671bab789262.slice/crio-c12cdbad2824519d5e891219645d208f94aa379befe074a591222824cebae084 WatchSource:0}: Error finding container c12cdbad2824519d5e891219645d208f94aa379befe074a591222824cebae084: Status 404 returned error can't find the container with id c12cdbad2824519d5e891219645d208f94aa379befe074a591222824cebae084 Oct 31 01:25:18.305863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:18.305785 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" event={"ID":"bfaf10a1-76a7-49ee-bd87-671bab789262","Type":"ContainerStarted","Data":"c12cdbad2824519d5e891219645d208f94aa379befe074a591222824cebae084"} Oct 31 01:25:21.323113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:21.323072 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" event={"ID":"bfaf10a1-76a7-49ee-bd87-671bab789262","Type":"ContainerStarted","Data":"a181d9be7f595a5258f72aa22ba6c1c49d164154da7bdb5adb02ac64ace2cab4"} Oct 31 01:25:21.323563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:21.323239 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:21.343429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:21.343379 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" podStartSLOduration=1.949935554 podStartE2EDuration="4.34336603s" totalImagesPullingTime="2.393430476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:25:21.342071288 +0000 UTC m=+321.722316529" watchObservedRunningTime="2026-10-31 01:25:21.34336603 +0000 UTC m=+321.723611279" Oct 31 01:25:21.490869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:21.490841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:21.632801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:21.632710 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:22.879274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:22.879242 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:23.014613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:23.014565 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:28.488351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:28.488318 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:28.658501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:28.658466 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:31.916938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:31.916904 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:32.059028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:32.058994 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:34.788208 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:25:34.788174 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:25:36.127857 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:36.127826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:36.279408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:36.279376 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:40.732992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:40.732960 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:40.880883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:40.880851 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:42.329176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:42.329141 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-8whxs" Oct 31 01:25:48.776187 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:48.776160 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:54.626049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:54.626018 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:25:54.798896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:25:54.798868 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:04.792512 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:26:04.792453 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:26:10.470896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.470847 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:10.529264 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.529232 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n8bs7"] Oct 31 01:26:10.532878 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.532858 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.537544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.537515 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 31 01:26:10.537648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.537584 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tlrg9\"" Oct 31 01:26:10.542566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.542534 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns"] Oct 31 01:26:10.545894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.545868 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.547528 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.547507 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n8bs7"] Oct 31 01:26:10.563976 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.563952 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns"] Oct 31 01:26:10.625436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.624822 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-h6p68\"" Oct 31 01:26:10.625436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.624966 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 01:26:10.648839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.648806 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:10.648994 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.648963 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2x7ns\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.649059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.648994 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmktf\" (UniqueName: \"kubernetes.io/projected/c88bc543-4551-48e1-9054-4ad3210fcd05-kube-api-access-wmktf\") pod \"kserve-controller-manager-776cc66d47-n8bs7\" (UID: \"c88bc543-4551-48e1-9054-4ad3210fcd05\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.649059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.649024 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vztq\" (UniqueName: \"kubernetes.io/projected/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-kube-api-access-4vztq\") pod \"llmisvc-controller-manager-869d5dcc5d-2x7ns\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.649158 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.649117 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c88bc543-4551-48e1-9054-4ad3210fcd05-cert\") pod \"kserve-controller-manager-776cc66d47-n8bs7\" (UID: \"c88bc543-4551-48e1-9054-4ad3210fcd05\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.749796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.749772 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2x7ns\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.749976 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.749811 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmktf\" (UniqueName: \"kubernetes.io/projected/c88bc543-4551-48e1-9054-4ad3210fcd05-kube-api-access-wmktf\") pod \"kserve-controller-manager-776cc66d47-n8bs7\" (UID: \"c88bc543-4551-48e1-9054-4ad3210fcd05\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.749976 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.749941 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vztq\" (UniqueName: \"kubernetes.io/projected/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-kube-api-access-4vztq\") pod \"llmisvc-controller-manager-869d5dcc5d-2x7ns\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.750046 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.749981 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c88bc543-4551-48e1-9054-4ad3210fcd05-cert\") pod \"kserve-controller-manager-776cc66d47-n8bs7\" (UID: \"c88bc543-4551-48e1-9054-4ad3210fcd05\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.752311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.752285 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2x7ns\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.752425 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.752398 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c88bc543-4551-48e1-9054-4ad3210fcd05-cert\") pod \"kserve-controller-manager-776cc66d47-n8bs7\" (UID: \"c88bc543-4551-48e1-9054-4ad3210fcd05\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.772025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.772002 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmktf\" (UniqueName: \"kubernetes.io/projected/c88bc543-4551-48e1-9054-4ad3210fcd05-kube-api-access-wmktf\") pod \"kserve-controller-manager-776cc66d47-n8bs7\" (UID: \"c88bc543-4551-48e1-9054-4ad3210fcd05\") " pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.779442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.779418 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vztq\" (UniqueName: \"kubernetes.io/projected/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-kube-api-access-4vztq\") pod \"llmisvc-controller-manager-869d5dcc5d-2x7ns\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.847583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.847558 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:10.861078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.861059 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:10.900940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:10.900890 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:11.066655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:11.066547 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:11.125925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:11.125903 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-n8bs7"] Oct 31 01:26:11.128587 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:26:11.128517 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88bc543_4551_48e1_9054_4ad3210fcd05.slice/crio-892e63fc82a05002430204bb7ffcec932dd3b888bedb858a36e5825dcd5b9481 WatchSource:0}: Error finding container 892e63fc82a05002430204bb7ffcec932dd3b888bedb858a36e5825dcd5b9481: Status 404 returned error can't find the container with id 892e63fc82a05002430204bb7ffcec932dd3b888bedb858a36e5825dcd5b9481 Oct 31 01:26:11.150506 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:11.150483 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns"] Oct 31 01:26:11.152858 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:26:11.152832 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod12173ba7_cd83_42c0_bd9b_a0b104b97ef1.slice/crio-0bb5fc61a954f5991a2ea62e8e66a34ea465a3f63c4a1097beba37da561eaeaf WatchSource:0}: Error finding container 0bb5fc61a954f5991a2ea62e8e66a34ea465a3f63c4a1097beba37da561eaeaf: Status 404 returned error can't find the container with id 0bb5fc61a954f5991a2ea62e8e66a34ea465a3f63c4a1097beba37da561eaeaf Oct 31 01:26:11.561332 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:11.561302 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" event={"ID":"12173ba7-cd83-42c0-bd9b-a0b104b97ef1","Type":"ContainerStarted","Data":"0bb5fc61a954f5991a2ea62e8e66a34ea465a3f63c4a1097beba37da561eaeaf"} Oct 31 01:26:11.562559 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:11.562537 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" event={"ID":"c88bc543-4551-48e1-9054-4ad3210fcd05","Type":"ContainerStarted","Data":"892e63fc82a05002430204bb7ffcec932dd3b888bedb858a36e5825dcd5b9481"} Oct 31 01:26:11.870745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:11.870634 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:12.079056 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:12.078972 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:12.279347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:12.279311 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:26:13.574917 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:13.574878 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" event={"ID":"c88bc543-4551-48e1-9054-4ad3210fcd05","Type":"ContainerStarted","Data":"565413bc0d53aaa61d75e281a1eb43a73acdb12f01ecaa6ee8785dd4f6c2d057"} Oct 31 01:26:13.575336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:13.575083 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:13.576891 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:13.576868 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" event={"ID":"12173ba7-cd83-42c0-bd9b-a0b104b97ef1","Type":"ContainerStarted","Data":"51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965"} Oct 31 01:26:13.577027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:13.577007 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:13.594152 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:13.594110 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" podStartSLOduration=1.5375567559999999 podStartE2EDuration="3.59409802s" totalImagesPullingTime="2.056541264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:13.592096202 +0000 UTC m=+373.972341426" watchObservedRunningTime="2026-10-31 01:26:13.59409802 +0000 UTC m=+373.974343250" Oct 31 01:26:13.609700 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:13.609653 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" podStartSLOduration=1.658583155 podStartE2EDuration="3.609643973s" totalImagesPullingTime="1.951060818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:13.60921546 +0000 UTC m=+373.989460702" watchObservedRunningTime="2026-10-31 01:26:13.609643973 +0000 UTC m=+373.989889196" Oct 31 01:26:34.786950 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:26:34.786914 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:26:44.583813 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:44.583711 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:26:44.586851 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:44.586826 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-n8bs7" Oct 31 01:26:48.830339 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:26:48.830310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:04.786944 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:27:04.786910 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:27:34.791877 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:27:34.791842 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:27:36.260132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:27:36.260092 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:27:48.932441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:27:48.932406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:04.788463 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:28:04.788431 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:28:16.447615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:28:16.447538 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:16.598483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:28:16.598452 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:28:34.786893 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:28:34.786859 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:28:48.927085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:28:48.927051 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:04.788299 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:29:04.788257 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:29:30.721208 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:29:30.721179 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:34.787885 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:29:34.787851 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:29:49.007323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:29:49.007242 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:29:59.806410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:29:59.806383 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:29:59.808149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:29:59.808124 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:29:59.811530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:29:59.811511 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:29:59.813189 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:29:59.813172 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:30:04.787803 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:30:04.787759 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:30:19.216011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:30:19.215975 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:34.787118 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:30:34.787077 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:30:49.054408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:30:49.054379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:30:50.551477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:30:50.551441 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:04.787293 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:31:04.787256 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:31:32.835903 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:31:32.835829 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:34.787421 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:31:34.787383 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:31:46.405934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:31:46.405902 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:49.086279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:31:49.086246 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:31:49.288067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:31:49.288033 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:02.225347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:32:02.225316 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:02.374036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:32:02.373997 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:04.787034 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:32:04.786998 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:32:34.787246 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:32:34.787208 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:32:49.123517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:32:49.123484 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:32:49.269985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:32:49.269954 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:33:04.788011 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:33:04.787970 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:33:34.787128 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:33:34.787080 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:33:49.277354 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:33:49.277321 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:04.791150 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:34:04.791117 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:34:25.841789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:34:25.841758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:34.788088 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:34:34.788043 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:34:49.269177 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:34:49.269145 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:34:59.838336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:34:59.838297 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:34:59.840182 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:34:59.840160 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:34:59.843201 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:34:59.843181 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:34:59.845073 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:34:59.845055 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:35:04.788109 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:35:04.788065 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:35:34.788196 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:35:34.788160 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:35:49.265350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:49.265296 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:51.823731 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.823687 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9nfbd"] Oct 31 01:35:51.827455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.827431 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:51.830325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.830300 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wvwnn\"" Oct 31 01:35:51.830773 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.830739 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 01:35:51.831088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.831069 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:51.838411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.838386 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9nfbd"] Oct 31 01:35:51.923170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.923142 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9nfbd"] Oct 31 01:35:51.959522 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.959489 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717bec7f-f324-4963-8490-083edabad375-config-file\") pod \"limitador-limitador-74dfcfb894-9nfbd\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:51.959698 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.959596 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnk8h\" (UniqueName: \"kubernetes.io/projected/717bec7f-f324-4963-8490-083edabad375-kube-api-access-gnk8h\") pod \"limitador-limitador-74dfcfb894-9nfbd\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:51.993687 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:51.993658 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:52.060603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.060568 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnk8h\" (UniqueName: \"kubernetes.io/projected/717bec7f-f324-4963-8490-083edabad375-kube-api-access-gnk8h\") pod \"limitador-limitador-74dfcfb894-9nfbd\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:52.060803 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.060652 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717bec7f-f324-4963-8490-083edabad375-config-file\") pod \"limitador-limitador-74dfcfb894-9nfbd\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:52.061263 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.061244 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717bec7f-f324-4963-8490-083edabad375-config-file\") pod \"limitador-limitador-74dfcfb894-9nfbd\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:52.072750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.072708 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnk8h\" (UniqueName: \"kubernetes.io/projected/717bec7f-f324-4963-8490-083edabad375-kube-api-access-gnk8h\") pod \"limitador-limitador-74dfcfb894-9nfbd\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:52.141345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.141285 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:52.147992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.147970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:52.420672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.420597 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9nfbd"] Oct 31 01:35:52.425266 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:35:52.425241 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717bec7f_f324_4963_8490_083edabad375.slice/crio-80a5116261f418e4d9152f221eeec2e91b8d8fb6fcd3c8f3c29d3c5a4afd2d31 WatchSource:0}: Error finding container 80a5116261f418e4d9152f221eeec2e91b8d8fb6fcd3c8f3c29d3c5a4afd2d31: Status 404 returned error can't find the container with id 80a5116261f418e4d9152f221eeec2e91b8d8fb6fcd3c8f3c29d3c5a4afd2d31 Oct 31 01:35:52.427062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.427044 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:35:52.664037 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.664006 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f4nls"] Oct 31 01:35:52.668701 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.668682 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f4nls" Oct 31 01:35:52.674370 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.674319 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z2xhp\"" Oct 31 01:35:52.676046 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.676021 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f4nls"] Oct 31 01:35:52.761651 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.761622 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:52.766818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.766788 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7452m\" (UniqueName: \"kubernetes.io/projected/8caa6007-ec50-4916-8470-4736ace520e9-kube-api-access-7452m\") pod \"authorino-57f7f8659f-f4nls\" (UID: \"8caa6007-ec50-4916-8470-4736ace520e9\") " pod="kuadrant-system/authorino-57f7f8659f-f4nls" Oct 31 01:35:52.867809 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.867774 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7452m\" (UniqueName: \"kubernetes.io/projected/8caa6007-ec50-4916-8470-4736ace520e9-kube-api-access-7452m\") pod \"authorino-57f7f8659f-f4nls\" (UID: \"8caa6007-ec50-4916-8470-4736ace520e9\") " pod="kuadrant-system/authorino-57f7f8659f-f4nls" Oct 31 01:35:52.879879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.879860 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7452m\" (UniqueName: \"kubernetes.io/projected/8caa6007-ec50-4916-8470-4736ace520e9-kube-api-access-7452m\") pod \"authorino-57f7f8659f-f4nls\" (UID: \"8caa6007-ec50-4916-8470-4736ace520e9\") " pod="kuadrant-system/authorino-57f7f8659f-f4nls" Oct 31 01:35:52.898536 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.898510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:35:52.989904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:52.989841 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f4nls" Oct 31 01:35:53.264258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:53.263912 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f4nls"] Oct 31 01:35:53.266913 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:35:53.266882 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8caa6007_ec50_4916_8470_4736ace520e9.slice/crio-3a179f8ffc66b3e09d4b52d84b1623cad54432c437ee68544a11ddcd977ecdc3 WatchSource:0}: Error finding container 3a179f8ffc66b3e09d4b52d84b1623cad54432c437ee68544a11ddcd977ecdc3: Status 404 returned error can't find the container with id 3a179f8ffc66b3e09d4b52d84b1623cad54432c437ee68544a11ddcd977ecdc3 Oct 31 01:35:53.277189 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:53.277161 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f4nls" event={"ID":"8caa6007-ec50-4916-8470-4736ace520e9","Type":"ContainerStarted","Data":"3a179f8ffc66b3e09d4b52d84b1623cad54432c437ee68544a11ddcd977ecdc3"} Oct 31 01:35:53.278251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:53.278231 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" event={"ID":"717bec7f-f324-4963-8490-083edabad375","Type":"ContainerStarted","Data":"80a5116261f418e4d9152f221eeec2e91b8d8fb6fcd3c8f3c29d3c5a4afd2d31"} Oct 31 01:35:54.284529 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:54.284496 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" event={"ID":"717bec7f-f324-4963-8490-083edabad375","Type":"ContainerStarted","Data":"2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166"} Oct 31 01:35:54.284833 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:54.284575 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:35:54.304429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:54.304388 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" podStartSLOduration=1.5444002829999999 podStartE2EDuration="3.304375421s" totalImagesPullingTime="1.759975138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:54.30276718 +0000 UTC m=+954.683012417" watchObservedRunningTime="2026-10-31 01:35:54.304375421 +0000 UTC m=+954.684620643" Oct 31 01:35:56.296983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:56.296953 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f4nls" event={"ID":"8caa6007-ec50-4916-8470-4736ace520e9","Type":"ContainerStarted","Data":"a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2"} Oct 31 01:35:56.313283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:35:56.313224 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-f4nls" podStartSLOduration=2.094963128 podStartE2EDuration="4.313209415s" totalImagesPullingTime="2.218246287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:35:56.312460079 +0000 UTC m=+956.692705322" watchObservedRunningTime="2026-10-31 01:35:56.313209415 +0000 UTC m=+956.693454637" Oct 31 01:36:00.387331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:00.387294 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f4nls"] Oct 31 01:36:00.387819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:00.387445 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-f4nls" podUID="8caa6007-ec50-4916-8470-4736ace520e9" containerName="authorino" containerID="cri-o://a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2" gracePeriod=30 Oct 31 01:36:00.638685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:00.638634 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f4nls" Oct 31 01:36:00.735368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:00.735337 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8caa6007-ec50-4916-8470-4736ace520e9-kube-api-access-7452m\" (UniqueName: \"kubernetes.io/projected/8caa6007-ec50-4916-8470-4736ace520e9-kube-api-access-7452m\") pod \"8caa6007-ec50-4916-8470-4736ace520e9\" (UID: \"8caa6007-ec50-4916-8470-4736ace520e9\") " Oct 31 01:36:00.737619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:00.737586 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8caa6007-ec50-4916-8470-4736ace520e9-kube-api-access-7452m" pod "8caa6007-ec50-4916-8470-4736ace520e9" (UID: "8caa6007-ec50-4916-8470-4736ace520e9"). InnerVolumeSpecName "kube-api-access-7452m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:00.836529 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:00.836498 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7452m\" (UniqueName: \"kubernetes.io/projected/8caa6007-ec50-4916-8470-4736ace520e9-kube-api-access-7452m\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:36:01.322168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.322132 5090 generic.go:356] "Generic (PLEG): container finished" podID="8caa6007-ec50-4916-8470-4736ace520e9" containerID="a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2" exitCode=0 Oct 31 01:36:01.322366 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.322181 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-f4nls" Oct 31 01:36:01.322366 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.322187 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f4nls" event={"ID":"8caa6007-ec50-4916-8470-4736ace520e9","Type":"ContainerDied","Data":"a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2"} Oct 31 01:36:01.322366 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.322226 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-f4nls" event={"ID":"8caa6007-ec50-4916-8470-4736ace520e9","Type":"ContainerDied","Data":"3a179f8ffc66b3e09d4b52d84b1623cad54432c437ee68544a11ddcd977ecdc3"} Oct 31 01:36:01.322366 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.322247 5090 scope.go:122] "RemoveContainer" containerID="a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2" Oct 31 01:36:01.334961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.334940 5090 scope.go:122] "RemoveContainer" containerID="a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2" Oct 31 01:36:01.335209 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:36:01.335190 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2\": container with ID starting with a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2 not found: ID does not exist" containerID="a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2" Oct 31 01:36:01.335256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.335219 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2"} err="failed to get container status \"a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2\": rpc error: code = NotFound desc = could not find container \"a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2\": container with ID starting with a25fb83d22c40ae963ecab165d685630c923024aa29c94f8abe6584349a6c7c2 not found: ID does not exist" Oct 31 01:36:01.345013 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.344988 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f4nls"] Oct 31 01:36:01.351874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.351855 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-f4nls"] Oct 31 01:36:01.603791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.603707 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:01.743284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.743259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:01.745666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:01.745642 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8caa6007-ec50-4916-8470-4736ace520e9" path="/var/lib/kubelet/pods/8caa6007-ec50-4916-8470-4736ace520e9/volumes" Oct 31 01:36:04.787587 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:36:04.787557 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:36:05.292275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:05.292251 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:36:05.705844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:05.705755 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9nfbd"] Oct 31 01:36:05.706124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:05.705999 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" podUID="717bec7f-f324-4963-8490-083edabad375" containerName="limitador" containerID="cri-o://2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166" gracePeriod=30 Oct 31 01:36:06.262157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.262106 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:36:06.278165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.278137 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717bec7f-f324-4963-8490-083edabad375-config-file\" (UniqueName: \"kubernetes.io/configmap/717bec7f-f324-4963-8490-083edabad375-config-file\") pod \"717bec7f-f324-4963-8490-083edabad375\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " Oct 31 01:36:06.278293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.278208 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717bec7f-f324-4963-8490-083edabad375-kube-api-access-gnk8h\" (UniqueName: \"kubernetes.io/projected/717bec7f-f324-4963-8490-083edabad375-kube-api-access-gnk8h\") pod \"717bec7f-f324-4963-8490-083edabad375\" (UID: \"717bec7f-f324-4963-8490-083edabad375\") " Oct 31 01:36:06.278500 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.278475 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717bec7f-f324-4963-8490-083edabad375-config-file" pod "717bec7f-f324-4963-8490-083edabad375" (UID: "717bec7f-f324-4963-8490-083edabad375"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:36:06.280343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.280321 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717bec7f-f324-4963-8490-083edabad375-kube-api-access-gnk8h" pod "717bec7f-f324-4963-8490-083edabad375" (UID: "717bec7f-f324-4963-8490-083edabad375"). InnerVolumeSpecName "kube-api-access-gnk8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:06.347654 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.347616 5090 generic.go:356] "Generic (PLEG): container finished" podID="717bec7f-f324-4963-8490-083edabad375" containerID="2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166" exitCode=0 Oct 31 01:36:06.347853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.347667 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" Oct 31 01:36:06.347853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.347691 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" event={"ID":"717bec7f-f324-4963-8490-083edabad375","Type":"ContainerDied","Data":"2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166"} Oct 31 01:36:06.347853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.347754 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9nfbd" event={"ID":"717bec7f-f324-4963-8490-083edabad375","Type":"ContainerDied","Data":"80a5116261f418e4d9152f221eeec2e91b8d8fb6fcd3c8f3c29d3c5a4afd2d31"} Oct 31 01:36:06.347853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.347776 5090 scope.go:122] "RemoveContainer" containerID="2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166" Oct 31 01:36:06.356668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.356647 5090 scope.go:122] "RemoveContainer" containerID="2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166" Oct 31 01:36:06.357012 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:36:06.356995 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166\": container with ID starting with 2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166 not found: ID does not exist" containerID="2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166" Oct 31 01:36:06.357079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.357018 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166"} err="failed to get container status \"2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166\": rpc error: code = NotFound desc = could not find container \"2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166\": container with ID starting with 2bb4cad7c9d1bb55300461feb80d43c8dff0b125c7a5319667aa654a24b30166 not found: ID does not exist" Oct 31 01:36:06.368162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.368142 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9nfbd"] Oct 31 01:36:06.372867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.372848 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9nfbd"] Oct 31 01:36:06.379729 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.379695 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnk8h\" (UniqueName: \"kubernetes.io/projected/717bec7f-f324-4963-8490-083edabad375-kube-api-access-gnk8h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:36:06.379805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:06.379732 5090 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/717bec7f-f324-4963-8490-083edabad375-config-file\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:36:07.745391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:07.745362 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717bec7f-f324-4963-8490-083edabad375" path="/var/lib/kubelet/pods/717bec7f-f324-4963-8490-083edabad375/volumes" Oct 31 01:36:08.555478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:08.555446 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:08.741408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:08.741378 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:09.313958 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.313928 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w"] Oct 31 01:36:09.314328 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.314313 5090 cpu_manager.go:414] "containerMap: removing container" podUID="8caa6007-ec50-4916-8470-4736ace520e9" containerName="authorino" Oct 31 01:36:09.314328 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.314324 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8caa6007-ec50-4916-8470-4736ace520e9" containerName="authorino" Oct 31 01:36:09.314399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.314336 5090 cpu_manager.go:414] "containerMap: removing container" podUID="717bec7f-f324-4963-8490-083edabad375" containerName="limitador" Oct 31 01:36:09.314399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.314341 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717bec7f-f324-4963-8490-083edabad375" containerName="limitador" Oct 31 01:36:09.314399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.314396 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="8caa6007-ec50-4916-8470-4736ace520e9" containerName="authorino" Oct 31 01:36:09.314489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.314403 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="717bec7f-f324-4963-8490-083edabad375" containerName="limitador" Oct 31 01:36:09.318870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.318851 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.325357 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.325255 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 01:36:09.325873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.325850 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 01:36:09.326490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.325966 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 01:36:09.326490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.326171 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tlc27\"" Oct 31 01:36:09.328116 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.328094 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w"] Oct 31 01:36:09.405171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.405145 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp85w\" (UniqueName: \"kubernetes.io/projected/d396d6a1-dd56-41ae-b577-7010cbcc590a-kube-api-access-wp85w\") pod \"postgres-7cdbf6dc44-68q6w\" (UID: \"d396d6a1-dd56-41ae-b577-7010cbcc590a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.405171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.405176 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d396d6a1-dd56-41ae-b577-7010cbcc590a-data\") pod \"postgres-7cdbf6dc44-68q6w\" (UID: \"d396d6a1-dd56-41ae-b577-7010cbcc590a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.442101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.442076 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:09.505672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.505643 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp85w\" (UniqueName: \"kubernetes.io/projected/d396d6a1-dd56-41ae-b577-7010cbcc590a-kube-api-access-wp85w\") pod \"postgres-7cdbf6dc44-68q6w\" (UID: \"d396d6a1-dd56-41ae-b577-7010cbcc590a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.505672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.505676 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d396d6a1-dd56-41ae-b577-7010cbcc590a-data\") pod \"postgres-7cdbf6dc44-68q6w\" (UID: \"d396d6a1-dd56-41ae-b577-7010cbcc590a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.506066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.506049 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d396d6a1-dd56-41ae-b577-7010cbcc590a-data\") pod \"postgres-7cdbf6dc44-68q6w\" (UID: \"d396d6a1-dd56-41ae-b577-7010cbcc590a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.518166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.518139 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp85w\" (UniqueName: \"kubernetes.io/projected/d396d6a1-dd56-41ae-b577-7010cbcc590a-kube-api-access-wp85w\") pod \"postgres-7cdbf6dc44-68q6w\" (UID: \"d396d6a1-dd56-41ae-b577-7010cbcc590a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.633246 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.633178 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:09.911454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:09.911383 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w"] Oct 31 01:36:09.914576 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:36:09.914551 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd396d6a1_dd56_41ae_b577_7010cbcc590a.slice/crio-9bf4b907d5d0b9ace59800e4a8f7d7fef3ad2a51d55944e0f14b0f3798d395fb WatchSource:0}: Error finding container 9bf4b907d5d0b9ace59800e4a8f7d7fef3ad2a51d55944e0f14b0f3798d395fb: Status 404 returned error can't find the container with id 9bf4b907d5d0b9ace59800e4a8f7d7fef3ad2a51d55944e0f14b0f3798d395fb Oct 31 01:36:10.375618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:10.375581 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" event={"ID":"d396d6a1-dd56-41ae-b577-7010cbcc590a","Type":"ContainerStarted","Data":"9bf4b907d5d0b9ace59800e4a8f7d7fef3ad2a51d55944e0f14b0f3798d395fb"} Oct 31 01:36:16.407920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:16.407887 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" event={"ID":"d396d6a1-dd56-41ae-b577-7010cbcc590a","Type":"ContainerStarted","Data":"5998c90f9cd89fcffed8c2d5b9b76831b4fdb63f116473e39b98c586545575d4"} Oct 31 01:36:16.408300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:16.408018 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:16.425052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:16.425010 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" podStartSLOduration=1.372063266 podStartE2EDuration="7.424997641s" totalImagesPullingTime="6.052934375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:16.424051856 +0000 UTC m=+976.804297113" watchObservedRunningTime="2026-10-31 01:36:16.424997641 +0000 UTC m=+976.805242864" Oct 31 01:36:22.449156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:22.449129 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-68q6w" Oct 31 01:36:30.141014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.140978 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:30.173132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.173103 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f756bf9c7-wkltg"] Oct 31 01:36:30.192642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.192617 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f756bf9c7-wkltg"] Oct 31 01:36:30.193789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.193638 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.199777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.199750 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 01:36:30.200121 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.200091 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4cjz9\"" Oct 31 01:36:30.200345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.200077 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 01:36:30.200466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.200386 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 01:36:30.282400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.282367 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-webhook-cert\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.282518 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.282405 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-metrics-certs\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.282518 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.282459 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5hc\" (UniqueName: \"kubernetes.io/projected/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-kube-api-access-2g5hc\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.283536 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.283513 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:30.382937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.382911 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-webhook-cert\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.382937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.382938 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-metrics-certs\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.383130 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.382965 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5hc\" (UniqueName: \"kubernetes.io/projected/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-kube-api-access-2g5hc\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.385434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.385413 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-webhook-cert\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.385535 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.385478 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-metrics-certs\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.395177 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.395130 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5hc\" (UniqueName: \"kubernetes.io/projected/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-kube-api-access-2g5hc\") pod \"maas-controller-7f756bf9c7-wkltg\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.433687 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.433669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:30.509841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.509818 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:30.601571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.601543 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:30.693126 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.693057 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8555557b5d-s7mt2"] Oct 31 01:36:30.697613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.697319 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.710684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.710657 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555557b5d-s7mt2"] Oct 31 01:36:30.786567 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.786529 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgm6\" (UniqueName: \"kubernetes.io/projected/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-kube-api-access-dwgm6\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.786752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.786582 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-metrics-certs\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.786752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.786665 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-webhook-cert\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.787490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.787466 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f756bf9c7-wkltg"] Oct 31 01:36:30.790415 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:36:30.790391 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e91f6e6_a096_44b5_b12d_2a35655bf9b1.slice/crio-4757bd1f6b41b3c31f11d312000a94c84788daf04be011d8ade7f3c08a4f0ec2 WatchSource:0}: Error finding container 4757bd1f6b41b3c31f11d312000a94c84788daf04be011d8ade7f3c08a4f0ec2: Status 404 returned error can't find the container with id 4757bd1f6b41b3c31f11d312000a94c84788daf04be011d8ade7f3c08a4f0ec2 Oct 31 01:36:30.887678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.887650 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgm6\" (UniqueName: \"kubernetes.io/projected/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-kube-api-access-dwgm6\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.887815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.887683 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-metrics-certs\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.887815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.887703 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-webhook-cert\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.890225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.890206 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-metrics-certs\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.890320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.890233 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-webhook-cert\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:30.899587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:30.899561 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgm6\" (UniqueName: \"kubernetes.io/projected/abaa3149-5e37-43e5-8d9d-1dc6a63b86b3-kube-api-access-dwgm6\") pod \"maas-controller-8555557b5d-s7mt2\" (UID: \"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3\") " pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:31.011400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:31.011373 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:31.285616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:31.285545 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8555557b5d-s7mt2"] Oct 31 01:36:31.288921 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:36:31.288894 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabaa3149_5e37_43e5_8d9d_1dc6a63b86b3.slice/crio-37521880e6408dda6245fbcbb4f206ce2409bde315e025dd728c7a5516784c0d WatchSource:0}: Error finding container 37521880e6408dda6245fbcbb4f206ce2409bde315e025dd728c7a5516784c0d: Status 404 returned error can't find the container with id 37521880e6408dda6245fbcbb4f206ce2409bde315e025dd728c7a5516784c0d Oct 31 01:36:31.485624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:31.485585 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555557b5d-s7mt2" event={"ID":"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3","Type":"ContainerStarted","Data":"37521880e6408dda6245fbcbb4f206ce2409bde315e025dd728c7a5516784c0d"} Oct 31 01:36:31.486837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:31.486812 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" event={"ID":"1e91f6e6-a096-44b5-b12d-2a35655bf9b1","Type":"ContainerStarted","Data":"4757bd1f6b41b3c31f11d312000a94c84788daf04be011d8ade7f3c08a4f0ec2"} Oct 31 01:36:34.792593 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:36:34.792532 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:36:35.465656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.465578 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:35.511221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.511186 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8555557b5d-s7mt2" event={"ID":"abaa3149-5e37-43e5-8d9d-1dc6a63b86b3","Type":"ContainerStarted","Data":"cdf146a5830ac7ab548df82d2d3d2271cc9b82979c3819f563b8fda3697e4c08"} Oct 31 01:36:35.511401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.511328 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:35.513055 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.513022 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" event={"ID":"1e91f6e6-a096-44b5-b12d-2a35655bf9b1","Type":"ContainerStarted","Data":"47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013"} Oct 31 01:36:35.513169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.513127 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:35.531614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.531557 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-8555557b5d-s7mt2" podStartSLOduration=1.706827409 podStartE2EDuration="5.531544621s" totalImagesPullingTime="3.824717212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:35.530072715 +0000 UTC m=+995.910317934" watchObservedRunningTime="2026-10-31 01:36:35.531544621 +0000 UTC m=+995.911789838" Oct 31 01:36:35.552560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.552523 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" podStartSLOduration=1.219937104 podStartE2EDuration="5.552511234s" totalImagesPullingTime="4.33257413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:35.551147224 +0000 UTC m=+995.931392447" watchObservedRunningTime="2026-10-31 01:36:35.552511234 +0000 UTC m=+995.932756456" Oct 31 01:36:35.619378 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:35.619347 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:36.810210 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.810179 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w"] Oct 31 01:36:36.813676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.813654 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:36.818437 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.818411 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w"] Oct 31 01:36:36.820196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.819955 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 01:36:36.820196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.819955 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 01:36:36.820854 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.820686 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 01:36:36.820854 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.820821 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-g2r5b\"" Oct 31 01:36:36.946951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.946917 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:36.946951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.946955 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:36.947150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.946975 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4svt\" (UniqueName: \"kubernetes.io/projected/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-kube-api-access-g4svt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:36.947150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:36.947065 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.004186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.004150 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:37.048576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.048547 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.048770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.048663 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.048770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.048696 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.048770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.048739 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4svt\" (UniqueName: \"kubernetes.io/projected/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-kube-api-access-g4svt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.049499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.049422 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.049619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.049538 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.051359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.051336 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.060688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.060640 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4svt\" (UniqueName: \"kubernetes.io/projected/3eb1a5b5-d544-440c-8838-f7fe2a93d8a6-kube-api-access-g4svt\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-qlv8w\" (UID: \"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.139738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.139686 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:37.207474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.207407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:37.422629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.422438 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w"] Oct 31 01:36:37.425502 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:36:37.425464 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb1a5b5_d544_440c_8838_f7fe2a93d8a6.slice/crio-bfcba2c2a4238f0a657ab0f5dc8d5187cbc512583b985c6e4861803c305d3af0 WatchSource:0}: Error finding container bfcba2c2a4238f0a657ab0f5dc8d5187cbc512583b985c6e4861803c305d3af0: Status 404 returned error can't find the container with id bfcba2c2a4238f0a657ab0f5dc8d5187cbc512583b985c6e4861803c305d3af0 Oct 31 01:36:37.523238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:37.523200 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" event={"ID":"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6","Type":"ContainerStarted","Data":"bfcba2c2a4238f0a657ab0f5dc8d5187cbc512583b985c6e4861803c305d3af0"} Oct 31 01:36:46.520057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:46.520020 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:46.520936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:46.520920 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8555557b5d-s7mt2" Oct 31 01:36:46.599181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:46.599143 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f756bf9c7-wkltg"] Oct 31 01:36:46.599471 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:46.599423 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" podUID="1e91f6e6-a096-44b5-b12d-2a35655bf9b1" containerName="manager" containerID="cri-o://47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013" gracePeriod=30 Oct 31 01:36:46.876608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:46.876581 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:47.042628 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.042551 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-webhook-cert\") pod \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " Oct 31 01:36:47.042628 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.042589 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-metrics-certs\") pod \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " Oct 31 01:36:47.042896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.042651 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-kube-api-access-2g5hc\" (UniqueName: \"kubernetes.io/projected/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-kube-api-access-2g5hc\") pod \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\" (UID: \"1e91f6e6-a096-44b5-b12d-2a35655bf9b1\") " Oct 31 01:36:47.045449 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.045418 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-webhook-cert" pod "1e91f6e6-a096-44b5-b12d-2a35655bf9b1" (UID: "1e91f6e6-a096-44b5-b12d-2a35655bf9b1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:36:47.045577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.045553 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-metrics-certs" pod "1e91f6e6-a096-44b5-b12d-2a35655bf9b1" (UID: "1e91f6e6-a096-44b5-b12d-2a35655bf9b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:36:47.045629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.045602 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-kube-api-access-2g5hc" pod "1e91f6e6-a096-44b5-b12d-2a35655bf9b1" (UID: "1e91f6e6-a096-44b5-b12d-2a35655bf9b1"). InnerVolumeSpecName "kube-api-access-2g5hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:36:47.144041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.144004 5090 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-webhook-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:36:47.144041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.144046 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:36:47.144274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.144064 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g5hc\" (UniqueName: \"kubernetes.io/projected/1e91f6e6-a096-44b5-b12d-2a35655bf9b1-kube-api-access-2g5hc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:36:47.589232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.589200 5090 generic.go:356] "Generic (PLEG): container finished" podID="1e91f6e6-a096-44b5-b12d-2a35655bf9b1" containerID="47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013" exitCode=0 Oct 31 01:36:47.589736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.589257 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" Oct 31 01:36:47.589736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.589286 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" event={"ID":"1e91f6e6-a096-44b5-b12d-2a35655bf9b1","Type":"ContainerDied","Data":"47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013"} Oct 31 01:36:47.589736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.589361 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f756bf9c7-wkltg" event={"ID":"1e91f6e6-a096-44b5-b12d-2a35655bf9b1","Type":"ContainerDied","Data":"4757bd1f6b41b3c31f11d312000a94c84788daf04be011d8ade7f3c08a4f0ec2"} Oct 31 01:36:47.589736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.589399 5090 scope.go:122] "RemoveContainer" containerID="47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013" Oct 31 01:36:47.600793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.600609 5090 scope.go:122] "RemoveContainer" containerID="47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013" Oct 31 01:36:47.600953 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:36:47.600936 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013\": container with ID starting with 47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013 not found: ID does not exist" containerID="47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013" Oct 31 01:36:47.601005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.600976 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013"} err="failed to get container status \"47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013\": rpc error: code = NotFound desc = could not find container \"47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013\": container with ID starting with 47abcecd98df16f3ea91fcbd16eff948b0138a347124ea5d2c0350c52bf4e013 not found: ID does not exist" Oct 31 01:36:47.613449 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.613425 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f756bf9c7-wkltg"] Oct 31 01:36:47.618545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.618523 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f756bf9c7-wkltg"] Oct 31 01:36:47.747097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:47.747064 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e91f6e6-a096-44b5-b12d-2a35655bf9b1" path="/var/lib/kubelet/pods/1e91f6e6-a096-44b5-b12d-2a35655bf9b1/volumes" Oct 31 01:36:49.310465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:49.310424 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:36:49.601275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:49.601194 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" event={"ID":"3eb1a5b5-d544-440c-8838-f7fe2a93d8a6","Type":"ContainerStarted","Data":"dd22276cc6ed4ed3a7576691ca7ac5f3fa413932a55dfefb52382623b04f794a"} Oct 31 01:36:49.601275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:49.601269 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:36:49.619588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:36:49.619545 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" podStartSLOduration=2.298620895 podStartE2EDuration="13.619536239s" totalImagesPullingTime="11.320915344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:49.619289834 +0000 UTC m=+1009.999535075" watchObservedRunningTime="2026-10-31 01:36:49.619536239 +0000 UTC m=+1009.999781468" Oct 31 01:37:00.611232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:00.611195 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-qlv8w" Oct 31 01:37:04.788058 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:37:04.788021 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:37:07.257277 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:07.257244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:07.408344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:07.408314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:08.471022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.470970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:08.540824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.540784 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7"] Oct 31 01:37:08.541360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.541337 5090 cpu_manager.go:414] "containerMap: removing container" podUID="1e91f6e6-a096-44b5-b12d-2a35655bf9b1" containerName="manager" Oct 31 01:37:08.541526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.541502 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e91f6e6-a096-44b5-b12d-2a35655bf9b1" containerName="manager" Oct 31 01:37:08.541682 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.541666 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e91f6e6-a096-44b5-b12d-2a35655bf9b1" containerName="manager" Oct 31 01:37:08.548702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.548682 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.552583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.552563 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 01:37:08.559745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.559690 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7"] Oct 31 01:37:08.620999 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.620944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:08.625062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.625041 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-metrics-certs\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.625192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.625104 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-maas-api-tls\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.625192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.625133 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2hz\" (UniqueName: \"kubernetes.io/projected/2430c32f-cc04-40da-b971-2ed47ecad26f-kube-api-access-xh2hz\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.625304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.625287 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2430c32f-cc04-40da-b971-2ed47ecad26f-openshift-service-ca\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.635497 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.635444 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 01:37:08.635744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.635698 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qjq8v\"" Oct 31 01:37:08.726424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.726338 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-metrics-certs\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.726424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.726412 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-maas-api-tls\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.726666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.726439 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2hz\" (UniqueName: \"kubernetes.io/projected/2430c32f-cc04-40da-b971-2ed47ecad26f-kube-api-access-xh2hz\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.726666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.726501 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2430c32f-cc04-40da-b971-2ed47ecad26f-openshift-service-ca\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.727466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.727440 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2430c32f-cc04-40da-b971-2ed47ecad26f-openshift-service-ca\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.728998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.728975 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-metrics-certs\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.729083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.729002 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-maas-api-tls\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.739129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.739109 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2hz\" (UniqueName: \"kubernetes.io/projected/2430c32f-cc04-40da-b971-2ed47ecad26f-kube-api-access-xh2hz\") pod \"maas-api-6c87bcc56b-47qd7\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:08.790916 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.790881 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:08.863096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:08.863057 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:09.145152 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:09.145003 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7"] Oct 31 01:37:09.147988 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:37:09.147960 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2430c32f_cc04_40da_b971_2ed47ecad26f.slice/crio-7392ab50123e837036b0b1996a7a620ebea6a127c9d7e2b009003219dba40a17 WatchSource:0}: Error finding container 7392ab50123e837036b0b1996a7a620ebea6a127c9d7e2b009003219dba40a17: Status 404 returned error can't find the container with id 7392ab50123e837036b0b1996a7a620ebea6a127c9d7e2b009003219dba40a17 Oct 31 01:37:09.705238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:09.705208 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" event={"ID":"2430c32f-cc04-40da-b971-2ed47ecad26f","Type":"ContainerStarted","Data":"7392ab50123e837036b0b1996a7a620ebea6a127c9d7e2b009003219dba40a17"} Oct 31 01:37:14.733282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:14.733249 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" event={"ID":"2430c32f-cc04-40da-b971-2ed47ecad26f","Type":"ContainerStarted","Data":"5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d"} Oct 31 01:37:14.733680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:14.733372 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:14.753348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:14.753304 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" podStartSLOduration=2.164676017 podStartE2EDuration="6.753291479s" totalImagesPullingTime="4.588615462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:14.752309399 +0000 UTC m=+1035.132554623" watchObservedRunningTime="2026-10-31 01:37:14.753291479 +0000 UTC m=+1035.133536701" Oct 31 01:37:20.036048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:19.994488 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8qmll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:48954->10.132.0.37:15021: read: connection reset by peer" start-of-body= Oct 31 01:37:20.036048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:19.994556 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" podUID="34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": read tcp 10.132.0.2:48954->10.132.0.37:15021: read: connection reset by peer" Oct 31 01:37:20.743031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:20.743005 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:20.766155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:20.766131 5090 generic.go:356] "Generic (PLEG): container finished" podID="34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24" containerID="3e7faebb2602296772d659583a6414ff49b9bcfd3577682df6bd57d4b77cd75e" exitCode=0 Oct 31 01:37:20.766300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:20.766208 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" event={"ID":"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24","Type":"ContainerDied","Data":"3e7faebb2602296772d659583a6414ff49b9bcfd3577682df6bd57d4b77cd75e"} Oct 31 01:37:20.766527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:20.766510 5090 scope.go:122] "RemoveContainer" containerID="3e7faebb2602296772d659583a6414ff49b9bcfd3577682df6bd57d4b77cd75e" Oct 31 01:37:20.767696 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:20.767669 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:37:20.767790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:20.767775 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:37:20.767827 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:20.767814 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:37:21.271653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.271622 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:21.427455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.427424 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:21.625070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.625001 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-gx592"] Oct 31 01:37:21.652346 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.652322 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-gx592"] Oct 31 01:37:21.652493 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.652428 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:21.655693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.655670 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z2xhp\"" Oct 31 01:37:21.656047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.656023 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 01:37:21.760484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.760456 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqn9\" (UniqueName: \"kubernetes.io/projected/303d8834-7d4d-4eb0-9448-606f9f64c6b0-kube-api-access-frqn9\") pod \"authorino-6889cbd8c4-gx592\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:21.760650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.760504 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/303d8834-7d4d-4eb0-9448-606f9f64c6b0-tls-cert\") pod \"authorino-6889cbd8c4-gx592\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:21.764286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.764261 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9"] Oct 31 01:37:21.798992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.798934 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" event={"ID":"34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24","Type":"ContainerStarted","Data":"c02e3cad782a420b3348e43765438325e340b1a0557729c3785a0ccec4552abf"} Oct 31 01:37:21.799368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.799290 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.804738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.802093 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9"] Oct 31 01:37:21.860988 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.860960 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b530d2f1-8c94-43ef-8cac-5237861063bb-maas-api-tls\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.861118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.861005 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b530d2f1-8c94-43ef-8cac-5237861063bb-openshift-service-ca\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.861118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.861091 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b530d2f1-8c94-43ef-8cac-5237861063bb-metrics-certs\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.861190 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.861134 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqn9\" (UniqueName: \"kubernetes.io/projected/303d8834-7d4d-4eb0-9448-606f9f64c6b0-kube-api-access-frqn9\") pod \"authorino-6889cbd8c4-gx592\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:21.861253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.861226 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/303d8834-7d4d-4eb0-9448-606f9f64c6b0-tls-cert\") pod \"authorino-6889cbd8c4-gx592\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:21.861381 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.861319 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5sn\" (UniqueName: \"kubernetes.io/projected/b530d2f1-8c94-43ef-8cac-5237861063bb-kube-api-access-sm5sn\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.864021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.864002 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/303d8834-7d4d-4eb0-9448-606f9f64c6b0-tls-cert\") pod \"authorino-6889cbd8c4-gx592\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:21.899884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.899798 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqn9\" (UniqueName: \"kubernetes.io/projected/303d8834-7d4d-4eb0-9448-606f9f64c6b0-kube-api-access-frqn9\") pod \"authorino-6889cbd8c4-gx592\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:21.962426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.962396 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b530d2f1-8c94-43ef-8cac-5237861063bb-openshift-service-ca\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.962585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.962436 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b530d2f1-8c94-43ef-8cac-5237861063bb-metrics-certs\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.962585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.962535 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5sn\" (UniqueName: \"kubernetes.io/projected/b530d2f1-8c94-43ef-8cac-5237861063bb-kube-api-access-sm5sn\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.962673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.962612 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b530d2f1-8c94-43ef-8cac-5237861063bb-maas-api-tls\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.963224 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.963164 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b530d2f1-8c94-43ef-8cac-5237861063bb-openshift-service-ca\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.965134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.965112 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b530d2f1-8c94-43ef-8cac-5237861063bb-maas-api-tls\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:21.965284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.965267 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b530d2f1-8c94-43ef-8cac-5237861063bb-metrics-certs\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:22.002741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.975524 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-gx592"] Oct 31 01:37:22.002741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.998866 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:37:22.002741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.994854 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8qmll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 01:37:22.002741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.998949 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" podUID="34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 01:37:22.002741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.999211 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:22.002741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:21.997386 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5sn\" (UniqueName: \"kubernetes.io/projected/b530d2f1-8c94-43ef-8cac-5237861063bb-kube-api-access-sm5sn\") pod \"maas-api-798bb5bfd4-v86t9\" (UID: \"b530d2f1-8c94-43ef-8cac-5237861063bb\") " pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:22.126351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:22.126323 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:22.328258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:22.328225 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-gx592"] Oct 31 01:37:22.333046 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:37:22.333019 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod303d8834_7d4d_4eb0_9448_606f9f64c6b0.slice/crio-b3f7d71e455763bbc00587efb43e4341252f356e917fd5fdec11d1912def467d WatchSource:0}: Error finding container b3f7d71e455763bbc00587efb43e4341252f356e917fd5fdec11d1912def467d: Status 404 returned error can't find the container with id b3f7d71e455763bbc00587efb43e4341252f356e917fd5fdec11d1912def467d Oct 31 01:37:22.443919 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:22.443891 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9"] Oct 31 01:37:22.446018 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:37:22.445991 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb530d2f1_8c94_43ef_8cac_5237861063bb.slice/crio-26bab29f99306320abe61d2ae0e81893e6f234ddc762335b60d3dcb87dd5acd0 WatchSource:0}: Error finding container 26bab29f99306320abe61d2ae0e81893e6f234ddc762335b60d3dcb87dd5acd0: Status 404 returned error can't find the container with id 26bab29f99306320abe61d2ae0e81893e6f234ddc762335b60d3dcb87dd5acd0 Oct 31 01:37:22.809378 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:22.809339 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-gx592" event={"ID":"303d8834-7d4d-4eb0-9448-606f9f64c6b0","Type":"ContainerStarted","Data":"b3f7d71e455763bbc00587efb43e4341252f356e917fd5fdec11d1912def467d"} Oct 31 01:37:22.810761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:22.810698 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" event={"ID":"b530d2f1-8c94-43ef-8cac-5237861063bb","Type":"ContainerStarted","Data":"26bab29f99306320abe61d2ae0e81893e6f234ddc762335b60d3dcb87dd5acd0"} Oct 31 01:37:22.997759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:22.996642 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8qmll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 01:37:22.997759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:22.996706 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" podUID="34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 01:37:23.819791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:23.819751 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-gx592" event={"ID":"303d8834-7d4d-4eb0-9448-606f9f64c6b0","Type":"ContainerStarted","Data":"f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251"} Oct 31 01:37:23.821847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:23.821815 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" event={"ID":"b530d2f1-8c94-43ef-8cac-5237861063bb","Type":"ContainerStarted","Data":"78d270ee1e813de46afdd16cdd62193cd51b608c2287cf926727da88685995c8"} Oct 31 01:37:23.822087 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:23.822067 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:23.843322 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:23.843279 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" podStartSLOduration=2.068082196 podStartE2EDuration="2.843267589s" totalImagesPullingTime="775.185393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:23.84189442 +0000 UTC m=+1044.222139644" watchObservedRunningTime="2026-10-31 01:37:23.843267589 +0000 UTC m=+1044.223512806" Oct 31 01:37:23.991207 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:23.991122 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8qmll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 01:37:23.991452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:23.991426 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" podUID="34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 01:37:24.723479 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.723448 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn"] Oct 31 01:37:24.726494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.726474 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:24.737468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.737444 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn"] Oct 31 01:37:24.811118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.811082 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ff1e38-7f31-4da0-9e4c-6683b8553a6d-cert\") pod \"llmisvc-controller-manager-6466c554d8-9kzpn\" (UID: \"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:24.811294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.811166 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djtz\" (UniqueName: \"kubernetes.io/projected/e2ff1e38-7f31-4da0-9e4c-6683b8553a6d-kube-api-access-7djtz\") pod \"llmisvc-controller-manager-6466c554d8-9kzpn\" (UID: \"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:24.827377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.827339 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-gx592" podUID="303d8834-7d4d-4eb0-9448-606f9f64c6b0" containerName="authorino" containerID="cri-o://f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251" gracePeriod=30 Oct 31 01:37:24.843160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.843123 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-gx592" podStartSLOduration=2.52031335 podStartE2EDuration="3.843109923s" totalImagesPullingTime="1.322796573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:24.84262316 +0000 UTC m=+1045.222868384" watchObservedRunningTime="2026-10-31 01:37:24.843109923 +0000 UTC m=+1045.223355145" Oct 31 01:37:24.912294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.912254 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ff1e38-7f31-4da0-9e4c-6683b8553a6d-cert\") pod \"llmisvc-controller-manager-6466c554d8-9kzpn\" (UID: \"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:24.912486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.912334 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djtz\" (UniqueName: \"kubernetes.io/projected/e2ff1e38-7f31-4da0-9e4c-6683b8553a6d-kube-api-access-7djtz\") pod \"llmisvc-controller-manager-6466c554d8-9kzpn\" (UID: \"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:24.914927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.914906 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2ff1e38-7f31-4da0-9e4c-6683b8553a6d-cert\") pod \"llmisvc-controller-manager-6466c554d8-9kzpn\" (UID: \"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:24.924877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.924854 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djtz\" (UniqueName: \"kubernetes.io/projected/e2ff1e38-7f31-4da0-9e4c-6683b8553a6d-kube-api-access-7djtz\") pod \"llmisvc-controller-manager-6466c554d8-9kzpn\" (UID: \"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:24.997823 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.993910 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8qmll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 01:37:24.997823 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:24.993969 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" podUID="34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 01:37:25.038166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.038141 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:25.120945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.120920 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:25.215024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.214991 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/303d8834-7d4d-4eb0-9448-606f9f64c6b0-tls-cert\" (UniqueName: \"kubernetes.io/secret/303d8834-7d4d-4eb0-9448-606f9f64c6b0-tls-cert\") pod \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " Oct 31 01:37:25.215217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.215085 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/303d8834-7d4d-4eb0-9448-606f9f64c6b0-kube-api-access-frqn9\" (UniqueName: \"kubernetes.io/projected/303d8834-7d4d-4eb0-9448-606f9f64c6b0-kube-api-access-frqn9\") pod \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\" (UID: \"303d8834-7d4d-4eb0-9448-606f9f64c6b0\") " Oct 31 01:37:25.217551 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.217523 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303d8834-7d4d-4eb0-9448-606f9f64c6b0-kube-api-access-frqn9" pod "303d8834-7d4d-4eb0-9448-606f9f64c6b0" (UID: "303d8834-7d4d-4eb0-9448-606f9f64c6b0"). InnerVolumeSpecName "kube-api-access-frqn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:25.227022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.227000 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/303d8834-7d4d-4eb0-9448-606f9f64c6b0-tls-cert" pod "303d8834-7d4d-4eb0-9448-606f9f64c6b0" (UID: "303d8834-7d4d-4eb0-9448-606f9f64c6b0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:25.316577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.316501 5090 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/303d8834-7d4d-4eb0-9448-606f9f64c6b0-tls-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:25.316577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.316529 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frqn9\" (UniqueName: \"kubernetes.io/projected/303d8834-7d4d-4eb0-9448-606f9f64c6b0-kube-api-access-frqn9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:25.339013 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.338981 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn"] Oct 31 01:37:25.341873 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:37:25.341849 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ff1e38_7f31_4da0_9e4c_6683b8553a6d.slice/crio-d2e7026e2acdda11f60cfcccd5a0571bb372903ad5e5456cf584911afa666571 WatchSource:0}: Error finding container d2e7026e2acdda11f60cfcccd5a0571bb372903ad5e5456cf584911afa666571: Status 404 returned error can't find the container with id d2e7026e2acdda11f60cfcccd5a0571bb372903ad5e5456cf584911afa666571 Oct 31 01:37:25.833592 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.833568 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" event={"ID":"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d","Type":"ContainerStarted","Data":"d2e7026e2acdda11f60cfcccd5a0571bb372903ad5e5456cf584911afa666571"} Oct 31 01:37:25.834656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.834630 5090 generic.go:356] "Generic (PLEG): container finished" podID="303d8834-7d4d-4eb0-9448-606f9f64c6b0" containerID="f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251" exitCode=0 Oct 31 01:37:25.834779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.834676 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-gx592" Oct 31 01:37:25.834779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.834680 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-gx592" event={"ID":"303d8834-7d4d-4eb0-9448-606f9f64c6b0","Type":"ContainerDied","Data":"f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251"} Oct 31 01:37:25.834779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.834740 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-gx592" event={"ID":"303d8834-7d4d-4eb0-9448-606f9f64c6b0","Type":"ContainerDied","Data":"b3f7d71e455763bbc00587efb43e4341252f356e917fd5fdec11d1912def467d"} Oct 31 01:37:25.834779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.834746 5090 scope.go:122] "RemoveContainer" containerID="f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251" Oct 31 01:37:25.855464 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.855447 5090 scope.go:122] "RemoveContainer" containerID="f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251" Oct 31 01:37:25.855741 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:37:25.855692 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251\": container with ID starting with f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251 not found: ID does not exist" containerID="f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251" Oct 31 01:37:25.855811 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.855739 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251"} err="failed to get container status \"f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251\": rpc error: code = NotFound desc = could not find container \"f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251\": container with ID starting with f3295550ef32825d84bcd3679f1877eeaaf0cab69e0af24c6e7673e62c61e251 not found: ID does not exist" Oct 31 01:37:25.891741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.891188 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-gx592"] Oct 31 01:37:25.891741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.891241 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-gx592"] Oct 31 01:37:25.993742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.988102 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-8qmll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 01:37:25.993742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:25.988161 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" podUID="34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 01:37:26.840837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:26.840797 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" event={"ID":"e2ff1e38-7f31-4da0-9e4c-6683b8553a6d","Type":"ContainerStarted","Data":"9cc509f7af4d654a72cccc81b8528e9034462961a1e1217af6934383bab43151"} Oct 31 01:37:26.841279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:26.840894 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:26.858897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:26.858848 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" podStartSLOduration=2.387365926 podStartE2EDuration="2.858832384s" totalImagesPullingTime="471.466458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:37:26.858659691 +0000 UTC m=+1047.238904925" watchObservedRunningTime="2026-10-31 01:37:26.858832384 +0000 UTC m=+1047.239077608" Oct 31 01:37:26.988745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:26.988705 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:37:26.988984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:26.988966 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:37:26.989672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:26.989655 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-8qmll" Oct 31 01:37:27.745806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:27.745775 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="303d8834-7d4d-4eb0-9448-606f9f64c6b0" path="/var/lib/kubelet/pods/303d8834-7d4d-4eb0-9448-606f9f64c6b0/volumes" Oct 31 01:37:29.832999 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:29.832972 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-798bb5bfd4-v86t9" Oct 31 01:37:29.881614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:29.881575 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7"] Oct 31 01:37:29.881815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:29.881789 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" podUID="2430c32f-cc04-40da-b971-2ed47ecad26f" containerName="maas-api" containerID="cri-o://5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d" gracePeriod=30 Oct 31 01:37:30.139304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.139281 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:30.263347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.263312 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2430c32f-cc04-40da-b971-2ed47ecad26f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2430c32f-cc04-40da-b971-2ed47ecad26f-openshift-service-ca\") pod \"2430c32f-cc04-40da-b971-2ed47ecad26f\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " Oct 31 01:37:30.263517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.263357 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2430c32f-cc04-40da-b971-2ed47ecad26f-kube-api-access-xh2hz\" (UniqueName: \"kubernetes.io/projected/2430c32f-cc04-40da-b971-2ed47ecad26f-kube-api-access-xh2hz\") pod \"2430c32f-cc04-40da-b971-2ed47ecad26f\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " Oct 31 01:37:30.263517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.263429 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-maas-api-tls\") pod \"2430c32f-cc04-40da-b971-2ed47ecad26f\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " Oct 31 01:37:30.263517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.263499 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-metrics-certs\") pod \"2430c32f-cc04-40da-b971-2ed47ecad26f\" (UID: \"2430c32f-cc04-40da-b971-2ed47ecad26f\") " Oct 31 01:37:30.263841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.263813 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2430c32f-cc04-40da-b971-2ed47ecad26f-openshift-service-ca" pod "2430c32f-cc04-40da-b971-2ed47ecad26f" (UID: "2430c32f-cc04-40da-b971-2ed47ecad26f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:37:30.265939 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.265916 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2430c32f-cc04-40da-b971-2ed47ecad26f-kube-api-access-xh2hz" pod "2430c32f-cc04-40da-b971-2ed47ecad26f" (UID: "2430c32f-cc04-40da-b971-2ed47ecad26f"). InnerVolumeSpecName "kube-api-access-xh2hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:30.266039 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.265960 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-maas-api-tls" pod "2430c32f-cc04-40da-b971-2ed47ecad26f" (UID: "2430c32f-cc04-40da-b971-2ed47ecad26f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:30.266039 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.265986 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-metrics-certs" pod "2430c32f-cc04-40da-b971-2ed47ecad26f" (UID: "2430c32f-cc04-40da-b971-2ed47ecad26f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:30.364283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.364248 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2430c32f-cc04-40da-b971-2ed47ecad26f-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:30.364283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.364273 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh2hz\" (UniqueName: \"kubernetes.io/projected/2430c32f-cc04-40da-b971-2ed47ecad26f-kube-api-access-xh2hz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:30.364283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.364285 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:30.364497 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.364295 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2430c32f-cc04-40da-b971-2ed47ecad26f-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:30.862783 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.862753 5090 generic.go:356] "Generic (PLEG): container finished" podID="2430c32f-cc04-40da-b971-2ed47ecad26f" containerID="5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d" exitCode=0 Oct 31 01:37:30.863226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.862798 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" Oct 31 01:37:30.863226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.862818 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" event={"ID":"2430c32f-cc04-40da-b971-2ed47ecad26f","Type":"ContainerDied","Data":"5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d"} Oct 31 01:37:30.863226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.862858 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7" event={"ID":"2430c32f-cc04-40da-b971-2ed47ecad26f","Type":"ContainerDied","Data":"7392ab50123e837036b0b1996a7a620ebea6a127c9d7e2b009003219dba40a17"} Oct 31 01:37:30.863226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.862878 5090 scope.go:122] "RemoveContainer" containerID="5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d" Oct 31 01:37:30.872485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.872470 5090 scope.go:122] "RemoveContainer" containerID="5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d" Oct 31 01:37:30.872827 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:37:30.872804 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d\": container with ID starting with 5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d not found: ID does not exist" containerID="5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d" Oct 31 01:37:30.872918 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.872837 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d"} err="failed to get container status \"5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d\": rpc error: code = NotFound desc = could not find container \"5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d\": container with ID starting with 5e0357e04550384d0674a413c1f1ac8043a0ac533f3a8f207fad987e8fac998d not found: ID does not exist" Oct 31 01:37:30.883891 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.883869 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7"] Oct 31 01:37:30.888848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:30.888826 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c87bcc56b-47qd7"] Oct 31 01:37:31.745454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:31.745426 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2430c32f-cc04-40da-b971-2ed47ecad26f" path="/var/lib/kubelet/pods/2430c32f-cc04-40da-b971-2ed47ecad26f/volumes" Oct 31 01:37:34.787822 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:37:34.787777 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:37:49.366111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:49.366075 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:57.847826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:57.847799 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9kzpn" Oct 31 01:37:57.894954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:57.894913 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns"] Oct 31 01:37:57.895137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:57.895094 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" podUID="12173ba7-cd83-42c0-bd9b-a0b104b97ef1" containerName="manager" containerID="cri-o://51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965" gracePeriod=30 Oct 31 01:37:58.143721 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:58.143696 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:37:58.196042 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:58.196013 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-kube-api-access-4vztq\" (UniqueName: \"kubernetes.io/projected/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-kube-api-access-4vztq\") pod \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " Oct 31 01:37:58.196186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:58.196057 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-cert\" (UniqueName: \"kubernetes.io/secret/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-cert\") pod \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\" (UID: \"12173ba7-cd83-42c0-bd9b-a0b104b97ef1\") " Oct 31 01:37:58.198192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:58.198169 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-cert" pod "12173ba7-cd83-42c0-bd9b-a0b104b97ef1" (UID: "12173ba7-cd83-42c0-bd9b-a0b104b97ef1"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:37:58.198315 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:58.198227 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-kube-api-access-4vztq" pod "12173ba7-cd83-42c0-bd9b-a0b104b97ef1" (UID: "12173ba7-cd83-42c0-bd9b-a0b104b97ef1"). InnerVolumeSpecName "kube-api-access-4vztq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:37:58.297530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:58.297503 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vztq\" (UniqueName: \"kubernetes.io/projected/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-kube-api-access-4vztq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:58.297530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:58.297528 5090 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12173ba7-cd83-42c0-bd9b-a0b104b97ef1-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:37:59.005154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.005121 5090 generic.go:356] "Generic (PLEG): container finished" podID="12173ba7-cd83-42c0-bd9b-a0b104b97ef1" containerID="51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965" exitCode=0 Oct 31 01:37:59.005154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.005159 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" event={"ID":"12173ba7-cd83-42c0-bd9b-a0b104b97ef1","Type":"ContainerDied","Data":"51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965"} Oct 31 01:37:59.005600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.005162 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" Oct 31 01:37:59.005600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.005197 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns" event={"ID":"12173ba7-cd83-42c0-bd9b-a0b104b97ef1","Type":"ContainerDied","Data":"0bb5fc61a954f5991a2ea62e8e66a34ea465a3f63c4a1097beba37da561eaeaf"} Oct 31 01:37:59.005600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.005218 5090 scope.go:122] "RemoveContainer" containerID="51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965" Oct 31 01:37:59.014487 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.014470 5090 scope.go:122] "RemoveContainer" containerID="51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965" Oct 31 01:37:59.014821 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:37:59.014797 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965\": container with ID starting with 51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965 not found: ID does not exist" containerID="51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965" Oct 31 01:37:59.014877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.014834 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965"} err="failed to get container status \"51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965\": rpc error: code = NotFound desc = could not find container \"51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965\": container with ID starting with 51aa79c00572cd7feb0a678e34505117a2d102f55391c17b08a6661bcc6d7965 not found: ID does not exist" Oct 31 01:37:59.024264 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.024241 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns"] Oct 31 01:37:59.029014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.028994 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2x7ns"] Oct 31 01:37:59.162926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.162898 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:59.331467 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.331391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:37:59.744685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:37:59.744656 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12173ba7-cd83-42c0-bd9b-a0b104b97ef1" path="/var/lib/kubelet/pods/12173ba7-cd83-42c0-bd9b-a0b104b97ef1/volumes" Oct 31 01:38:04.341159 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:04.341110 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:04.485940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:04.485911 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:04.793616 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:38:04.793575 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:38:08.920924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:08.920890 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:28.837916 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:28.837878 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:29.354049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:29.354009 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:30.670624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:30.670585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:30.818752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:30.818704 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:34.546826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:34.546795 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:34.788373 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:38:34.788336 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:38:36.107909 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:36.107868 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:37.003632 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:37.003597 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:38.135498 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:38.135464 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:38.748869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:38.748842 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:40.701399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:40.701290 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:41.450992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:41.450964 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:43.421134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:43.421097 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:38:49.395846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:38:49.395814 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:00.374923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:00.374887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:00.521310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:00.521282 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:02.062936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:02.062894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:02.213927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:02.213893 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:02.764634 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:02.764602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:04.788663 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:39:04.788621 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:39:14.750241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.750200 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6"] Oct 31 01:39:14.751554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.751526 5090 cpu_manager.go:414] "containerMap: removing container" podUID="2430c32f-cc04-40da-b971-2ed47ecad26f" containerName="maas-api" Oct 31 01:39:14.751683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.751553 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2430c32f-cc04-40da-b971-2ed47ecad26f" containerName="maas-api" Oct 31 01:39:14.751683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.751601 5090 cpu_manager.go:414] "containerMap: removing container" podUID="12173ba7-cd83-42c0-bd9b-a0b104b97ef1" containerName="manager" Oct 31 01:39:14.751683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.751611 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12173ba7-cd83-42c0-bd9b-a0b104b97ef1" containerName="manager" Oct 31 01:39:14.751683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.751652 5090 cpu_manager.go:414] "containerMap: removing container" podUID="303d8834-7d4d-4eb0-9448-606f9f64c6b0" containerName="authorino" Oct 31 01:39:14.751683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.751659 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="303d8834-7d4d-4eb0-9448-606f9f64c6b0" containerName="authorino" Oct 31 01:39:14.752072 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.752052 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="2430c32f-cc04-40da-b971-2ed47ecad26f" containerName="maas-api" Oct 31 01:39:14.752154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.752083 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="12173ba7-cd83-42c0-bd9b-a0b104b97ef1" containerName="manager" Oct 31 01:39:14.752154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.752097 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="303d8834-7d4d-4eb0-9448-606f9f64c6b0" containerName="authorino" Oct 31 01:39:14.758027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.757997 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.765391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.765361 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6"] Oct 31 01:39:14.871196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871163 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871197 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871227 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3621e1c-e276-4304-a11d-8a44a339ea04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871384 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871472 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871538 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871634 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871556 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871634 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871579 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.871634 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.871598 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrmmg\" (UniqueName: \"kubernetes.io/projected/a3621e1c-e276-4304-a11d-8a44a339ea04-kube-api-access-hrmmg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.972887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.972854 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3621e1c-e276-4304-a11d-8a44a339ea04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973034 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.972896 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973034 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.972942 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973034 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.972986 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973034 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973009 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973042 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973068 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrmmg\" (UniqueName: \"kubernetes.io/projected/a3621e1c-e276-4304-a11d-8a44a339ea04-kube-api-access-hrmmg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973116 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973147 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973570 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973639 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3621e1c-e276-4304-a11d-8a44a339ea04-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973643 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973689 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.973739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.973707 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.975593 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.975572 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.975731 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.975699 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.995035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.995017 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrmmg\" (UniqueName: \"kubernetes.io/projected/a3621e1c-e276-4304-a11d-8a44a339ea04-kube-api-access-hrmmg\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:14.995134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:14.995112 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3621e1c-e276-4304-a11d-8a44a339ea04-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9vpv6\" (UID: \"a3621e1c-e276-4304-a11d-8a44a339ea04\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:15.074002 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:15.073919 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:15.354913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:15.354864 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6"] Oct 31 01:39:15.357209 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:39:15.357183 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3621e1c_e276_4304_a11d_8a44a339ea04.slice/crio-b854649ad1fd9fe8fe22f75006e50690017eeca9eeaae1ba7a7bd86da7753c2e WatchSource:0}: Error finding container b854649ad1fd9fe8fe22f75006e50690017eeca9eeaae1ba7a7bd86da7753c2e: Status 404 returned error can't find the container with id b854649ad1fd9fe8fe22f75006e50690017eeca9eeaae1ba7a7bd86da7753c2e Oct 31 01:39:15.359284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:15.359254 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:39:15.359345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:15.359336 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:39:15.359380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:15.359366 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:39:15.388123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:15.388098 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" event={"ID":"a3621e1c-e276-4304-a11d-8a44a339ea04","Type":"ContainerStarted","Data":"b854649ad1fd9fe8fe22f75006e50690017eeca9eeaae1ba7a7bd86da7753c2e"} Oct 31 01:39:16.403655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:16.403621 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" event={"ID":"a3621e1c-e276-4304-a11d-8a44a339ea04","Type":"ContainerStarted","Data":"f4581d3497b3ffaafe7117422608cc79373ec93e6b2127beb59d920ed951dc9e"} Oct 31 01:39:16.427241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:16.427183 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podStartSLOduration=2.42716874 podStartE2EDuration="2.42716874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:16.425851117 +0000 UTC m=+1156.806096340" watchObservedRunningTime="2026-10-31 01:39:16.42716874 +0000 UTC m=+1156.807414036" Oct 31 01:39:17.093741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:17.086255 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:17.093741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:17.086376 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:39:17.093741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:17.086421 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podUID="a3621e1c-e276-4304-a11d-8a44a339ea04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 01:39:18.093741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:18.088081 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:39:18.093741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:18.088147 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podUID="a3621e1c-e276-4304-a11d-8a44a339ea04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 01:39:19.093742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:19.087063 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:39:19.093742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:19.087121 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podUID="a3621e1c-e276-4304-a11d-8a44a339ea04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 01:39:20.097742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:20.096182 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:39:20.097742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:20.096246 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podUID="a3621e1c-e276-4304-a11d-8a44a339ea04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 01:39:21.094741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:21.090120 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:39:21.094741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:21.090194 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podUID="a3621e1c-e276-4304-a11d-8a44a339ea04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 01:39:22.075296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:22.075257 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 31 01:39:22.075705 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:22.075341 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podUID="a3621e1c-e276-4304-a11d-8a44a339ea04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 31 01:39:23.464461 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:23.464417 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:23.464937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:23.464828 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:23.465616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:23.465598 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" Oct 31 01:39:34.787698 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:39:34.787659 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:39:41.056438 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:41.056403 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:41.234872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:41.234838 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:41.391870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:41.391790 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:41.584757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:41.584704 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:41.758400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:41.758358 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:41.941477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:41.941442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:42.156299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:42.156215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:42.294340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:42.294306 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:43.075785 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:43.075749 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:43.615271 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:43.615234 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:43.760656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:43.760626 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:44.146380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:44.146345 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:44.301901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:44.301875 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:45.368820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:45.368787 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:45.519604 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:45.519573 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:45.657867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:45.657793 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:45.924327 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:45.924237 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:46.087139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:46.087109 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:46.241132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:46.241052 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:46.665801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:46.665769 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:46.850016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:46.849984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:47.689781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:47.689751 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:47.999872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:47.999837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:48.141686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.141647 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:48.359001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.358930 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9"] Oct 31 01:39:48.362824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.362804 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.368770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.368740 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6198e156-serving-cert\"" Oct 31 01:39:48.374694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.374671 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9"] Oct 31 01:39:48.465662 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.465629 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-metrics-certs\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.465859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.465695 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a45ee50-e1ea-425a-8795-9fba0c61c738-openshift-service-ca\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.465859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.465806 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-maas-api-tls\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.465951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.465868 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566hl\" (UniqueName: \"kubernetes.io/projected/5a45ee50-e1ea-425a-8795-9fba0c61c738-kube-api-access-566hl\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.503801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.503775 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:48.566315 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.566290 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566hl\" (UniqueName: \"kubernetes.io/projected/5a45ee50-e1ea-425a-8795-9fba0c61c738-kube-api-access-566hl\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.566489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.566385 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-metrics-certs\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.566489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.566470 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a45ee50-e1ea-425a-8795-9fba0c61c738-openshift-service-ca\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.566623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.566497 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-maas-api-tls\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.567324 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.567293 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a45ee50-e1ea-425a-8795-9fba0c61c738-openshift-service-ca\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.569074 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.569051 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-maas-api-tls\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.569165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.569117 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-metrics-certs\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.577620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.577601 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566hl\" (UniqueName: \"kubernetes.io/projected/5a45ee50-e1ea-425a-8795-9fba0c61c738-kube-api-access-566hl\") pod \"maas-api-e2e-ait-6198e156-68dfdb8584-kstb9\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.645191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.645095 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:48.677086 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.677059 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:48.955422 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:48.955342 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9"] Oct 31 01:39:48.958285 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:39:48.958250 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a45ee50_e1ea_425a_8795_9fba0c61c738.slice/crio-15ba3c72c88bec88846a9b0edb5523768d143d0f584a3213dd0112507f2ea3c3 WatchSource:0}: Error finding container 15ba3c72c88bec88846a9b0edb5523768d143d0f584a3213dd0112507f2ea3c3: Status 404 returned error can't find the container with id 15ba3c72c88bec88846a9b0edb5523768d143d0f584a3213dd0112507f2ea3c3 Oct 31 01:39:49.253432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.253404 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz"] Oct 31 01:39:49.258276 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.258250 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.267143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.267099 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6e7ff10a-serving-cert\"" Oct 31 01:39:49.271269 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.271214 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz"] Oct 31 01:39:49.273066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.273042 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvdvc\" (UniqueName: \"kubernetes.io/projected/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-kube-api-access-lvdvc\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.273185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.273091 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-maas-api-tls\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.273241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.273227 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-openshift-service-ca\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.273278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.273252 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-metrics-certs\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.349088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.349054 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:49.374155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.374127 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvdvc\" (UniqueName: \"kubernetes.io/projected/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-kube-api-access-lvdvc\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.374311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.374179 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-maas-api-tls\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.374311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.374269 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-openshift-service-ca\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.374311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.374291 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-metrics-certs\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.375013 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.374988 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-openshift-service-ca\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.376558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.376536 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-maas-api-tls\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.376668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.376653 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-metrics-certs\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.388655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.388637 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvdvc\" (UniqueName: \"kubernetes.io/projected/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-kube-api-access-lvdvc\") pod \"maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.496003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.495977 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:49.573446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.573370 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" event={"ID":"5a45ee50-e1ea-425a-8795-9fba0c61c738","Type":"ContainerStarted","Data":"1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426"} Oct 31 01:39:49.573446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.573429 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" event={"ID":"5a45ee50-e1ea-425a-8795-9fba0c61c738","Type":"ContainerStarted","Data":"15ba3c72c88bec88846a9b0edb5523768d143d0f584a3213dd0112507f2ea3c3"} Oct 31 01:39:49.573883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.573557 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:49.573883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.573581 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:49.594253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.593483 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" podStartSLOduration=1.116685319 podStartE2EDuration="1.593466558s" totalImagesPullingTime="476.781239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:49.592168084 +0000 UTC m=+1189.972413306" watchObservedRunningTime="2026-10-31 01:39:49.593466558 +0000 UTC m=+1189.973711779" Oct 31 01:39:49.881520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:49.881415 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz"] Oct 31 01:39:49.890922 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:39:49.890156 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ede1c1_bb24_450b_8dc3_0e56c847c3d0.slice/crio-ccb1f7b956f5a6e65abf0f1303feedb1fcfe14e4c3e2c4ec1ceaf9b8c4283ab4 WatchSource:0}: Error finding container ccb1f7b956f5a6e65abf0f1303feedb1fcfe14e4c3e2c4ec1ceaf9b8c4283ab4: Status 404 returned error can't find the container with id ccb1f7b956f5a6e65abf0f1303feedb1fcfe14e4c3e2c4ec1ceaf9b8c4283ab4 Oct 31 01:39:50.239221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.239145 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6"] Oct 31 01:39:50.244861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.244835 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.253388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.253351 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-aba34286-serving-cert\"" Oct 31 01:39:50.260848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.260816 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6"] Oct 31 01:39:50.285276 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.285247 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-metrics-certs\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.285441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.285301 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-openshift-service-ca\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.285441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.285398 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2twv\" (UniqueName: \"kubernetes.io/projected/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-kube-api-access-l2twv\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.285549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.285452 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-maas-api-tls\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.324892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.324846 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:50.386315 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.386273 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-maas-api-tls\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.386520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.386375 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-metrics-certs\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.386520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.386422 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-openshift-service-ca\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.386520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.386467 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2twv\" (UniqueName: \"kubernetes.io/projected/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-kube-api-access-l2twv\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.387124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.387090 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-openshift-service-ca\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.388951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.388929 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-maas-api-tls\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.389044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.388984 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-metrics-certs\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.398395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.398373 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2twv\" (UniqueName: \"kubernetes.io/projected/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-kube-api-access-l2twv\") pod \"maas-api-e2e-migrate-aba34286-85464cd89-dvgp6\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.479114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.479092 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:50.565310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.565270 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:50.580825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.580784 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" event={"ID":"67ede1c1-bb24-450b-8dc3-0e56c847c3d0","Type":"ContainerStarted","Data":"81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f"} Oct 31 01:39:50.580987 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.580850 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" event={"ID":"67ede1c1-bb24-450b-8dc3-0e56c847c3d0","Type":"ContainerStarted","Data":"ccb1f7b956f5a6e65abf0f1303feedb1fcfe14e4c3e2c4ec1ceaf9b8c4283ab4"} Oct 31 01:39:50.640858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.640831 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:50.780872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.780841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:50.846789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.846740 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" podStartSLOduration=1.3040440150000001 podStartE2EDuration="1.846701448s" totalImagesPullingTime="542.657433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:50.602922841 +0000 UTC m=+1190.983168085" watchObservedRunningTime="2026-10-31 01:39:50.846701448 +0000 UTC m=+1191.226946673" Oct 31 01:39:50.847150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.847131 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6"] Oct 31 01:39:50.849496 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:39:50.849465 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe288d0_ef2a_4b1a_95eb_fc1c47bacb3a.slice/crio-13c015306496dbe4c81474ca5c859f0367eee0848d04d4b113a76f2dab09f958 WatchSource:0}: Error finding container 13c015306496dbe4c81474ca5c859f0367eee0848d04d4b113a76f2dab09f958: Status 404 returned error can't find the container with id 13c015306496dbe4c81474ca5c859f0367eee0848d04d4b113a76f2dab09f958 Oct 31 01:39:50.995932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:50.995896 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:51.163568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.163477 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:51.491628 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.491600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:51.598114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.598076 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" event={"ID":"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a","Type":"ContainerStarted","Data":"0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82"} Oct 31 01:39:51.598336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.598169 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:51.598336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.598200 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" event={"ID":"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a","Type":"ContainerStarted","Data":"13c015306496dbe4c81474ca5c859f0367eee0848d04d4b113a76f2dab09f958"} Oct 31 01:39:51.598336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.598254 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:51.619371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.619314 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" podStartSLOduration=1.100362055 podStartE2EDuration="1.619300564s" totalImagesPullingTime="518.938509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:51.616529334 +0000 UTC m=+1191.996774557" watchObservedRunningTime="2026-10-31 01:39:51.619300564 +0000 UTC m=+1191.999545786" Oct 31 01:39:51.647557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:51.647527 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:52.067078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:52.067043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:52.784307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:52.784268 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:53.626249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:53.626207 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:54.238415 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:54.238384 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:54.386476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:54.386449 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:54.523416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:54.523382 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:55.141735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:55.141680 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:55.476025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:55.475944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:55.586332 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:55.586307 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:39:55.643173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:55.643141 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:55.931431 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:55.931376 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:56.814380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:56.814349 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:57.608414 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.608387 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:39:57.608888 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.608871 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:39:57.627228 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.627200 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5"] Oct 31 01:39:57.632601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.632581 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.644298 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.642859 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4db52a49-serving-cert\"" Oct 31 01:39:57.647057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.647036 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5"] Oct 31 01:39:57.668422 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.668392 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-metrics-certs\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.668559 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.668437 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz8kk\" (UniqueName: \"kubernetes.io/projected/c17ea991-26db-4514-8f98-d658c1f4ec3b-kube-api-access-tz8kk\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.668559 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.668542 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ea991-26db-4514-8f98-d658c1f4ec3b-openshift-service-ca\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.668694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.668582 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-maas-api-tls\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.727703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.727675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:57.769360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.769329 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-metrics-certs\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.769532 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.769365 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz8kk\" (UniqueName: \"kubernetes.io/projected/c17ea991-26db-4514-8f98-d658c1f4ec3b-kube-api-access-tz8kk\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.769532 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.769416 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ea991-26db-4514-8f98-d658c1f4ec3b-openshift-service-ca\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.769738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.769536 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-maas-api-tls\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.770113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.770091 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ea991-26db-4514-8f98-d658c1f4ec3b-openshift-service-ca\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.772026 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.772001 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-metrics-certs\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.772852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.772824 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-maas-api-tls\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.780920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.780899 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz8kk\" (UniqueName: \"kubernetes.io/projected/c17ea991-26db-4514-8f98-d658c1f4ec3b-kube-api-access-tz8kk\") pod \"maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:57.856464 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.856434 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:57.971455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:57.971373 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:58.286962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:58.286915 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5"] Oct 31 01:39:58.290605 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:39:58.290576 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17ea991_26db_4514_8f98_d658c1f4ec3b.slice/crio-00cd8884699725fc2f8fbc4617c012b866d47e91a49be51715d7f97cfaa08503 WatchSource:0}: Error finding container 00cd8884699725fc2f8fbc4617c012b866d47e91a49be51715d7f97cfaa08503: Status 404 returned error can't find the container with id 00cd8884699725fc2f8fbc4617c012b866d47e91a49be51715d7f97cfaa08503 Oct 31 01:39:58.443285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:58.443252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:58.648932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:58.648842 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" event={"ID":"c17ea991-26db-4514-8f98-d658c1f4ec3b","Type":"ContainerStarted","Data":"00cd8884699725fc2f8fbc4617c012b866d47e91a49be51715d7f97cfaa08503"} Oct 31 01:39:59.206331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.206034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:39:59.657414 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.657379 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" event={"ID":"c17ea991-26db-4514-8f98-d658c1f4ec3b","Type":"ContainerStarted","Data":"bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0"} Oct 31 01:39:59.657596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.657463 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:39:59.678335 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.678282 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" podStartSLOduration=1.8111139 podStartE2EDuration="2.678264393s" totalImagesPullingTime="867.150493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:39:59.676362192 +0000 UTC m=+1200.056607408" watchObservedRunningTime="2026-10-31 01:39:59.678264393 +0000 UTC m=+1200.058509618" Oct 31 01:39:59.871367 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.871336 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:39:59.881673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.881618 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:39:59.885028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.884996 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:39:59.891015 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:39:59.890990 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:40:00.058656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:00.058615 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:00.888560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:00.888525 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:01.560149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:01.560115 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:02.443109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:02.443073 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:03.295656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:03.295621 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:04.060522 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:04.060493 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:04.649356 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:04.649322 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:04.788375 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:04.788340 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:40:05.334139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:05.334101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:05.668344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:05.668267 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:40:06.259538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:06.259511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:07.144609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:07.144560 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:08.002755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:08.002674 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:08.753355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:08.753316 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:09.541702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:09.541654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:10.148186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:10.148100 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:10.830458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:10.830423 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:11.440431 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:11.440396 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:12.136292 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:12.136268 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:12.753649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:12.753621 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:13.457411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:13.457381 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:14.155397 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:14.155364 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:14.862349 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:14.862318 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:15.451539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:15.451507 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:15.784126 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:15.784091 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:15.895220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:15.895181 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz"] Oct 31 01:40:15.895738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:15.895681 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" podUID="67ede1c1-bb24-450b-8dc3-0e56c847c3d0" containerName="maas-api" containerID="cri-o://81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f" gracePeriod=30 Oct 31 01:40:15.929848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:15.929819 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:16.126491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.126456 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:16.164892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.164871 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:40:16.259990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.259958 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-openshift-service-ca\") pod \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " Oct 31 01:40:16.260151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.260033 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-metrics-certs\") pod \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " Oct 31 01:40:16.260151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.260118 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-kube-api-access-lvdvc\" (UniqueName: \"kubernetes.io/projected/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-kube-api-access-lvdvc\") pod \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " Oct 31 01:40:16.260247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.260219 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-maas-api-tls\") pod \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\" (UID: \"67ede1c1-bb24-450b-8dc3-0e56c847c3d0\") " Oct 31 01:40:16.260405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.260384 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-openshift-service-ca" pod "67ede1c1-bb24-450b-8dc3-0e56c847c3d0" (UID: "67ede1c1-bb24-450b-8dc3-0e56c847c3d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:16.260547 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.260530 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:16.262383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.262356 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-metrics-certs" pod "67ede1c1-bb24-450b-8dc3-0e56c847c3d0" (UID: "67ede1c1-bb24-450b-8dc3-0e56c847c3d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:16.262383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.262374 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-maas-api-tls" pod "67ede1c1-bb24-450b-8dc3-0e56c847c3d0" (UID: "67ede1c1-bb24-450b-8dc3-0e56c847c3d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:16.262550 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.262532 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-kube-api-access-lvdvc" pod "67ede1c1-bb24-450b-8dc3-0e56c847c3d0" (UID: "67ede1c1-bb24-450b-8dc3-0e56c847c3d0"). InnerVolumeSpecName "kube-api-access-lvdvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:16.350307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.350219 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:16.361858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.361826 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:16.361858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.361851 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvdvc\" (UniqueName: \"kubernetes.io/projected/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-kube-api-access-lvdvc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:16.361858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.361862 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67ede1c1-bb24-450b-8dc3-0e56c847c3d0-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:16.491690 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.491657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:16.746859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.746831 5090 generic.go:356] "Generic (PLEG): container finished" podID="67ede1c1-bb24-450b-8dc3-0e56c847c3d0" containerID="81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f" exitCode=0 Oct 31 01:40:16.747064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.746883 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" Oct 31 01:40:16.747064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.746912 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" event={"ID":"67ede1c1-bb24-450b-8dc3-0e56c847c3d0","Type":"ContainerDied","Data":"81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f"} Oct 31 01:40:16.747064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.746961 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz" event={"ID":"67ede1c1-bb24-450b-8dc3-0e56c847c3d0","Type":"ContainerDied","Data":"ccb1f7b956f5a6e65abf0f1303feedb1fcfe14e4c3e2c4ec1ceaf9b8c4283ab4"} Oct 31 01:40:16.747064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.746983 5090 scope.go:122] "RemoveContainer" containerID="81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f" Oct 31 01:40:16.757174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.757155 5090 scope.go:122] "RemoveContainer" containerID="81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f" Oct 31 01:40:16.757485 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:16.757461 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f\": container with ID starting with 81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f not found: ID does not exist" containerID="81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f" Oct 31 01:40:16.757575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.757497 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f"} err="failed to get container status \"81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f\": rpc error: code = NotFound desc = could not find container \"81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f\": container with ID starting with 81e8c05a9d7b7330146c1f44cec63e334be81b1a70b54f94072883c9bc5fa26f not found: ID does not exist" Oct 31 01:40:16.768595 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.768574 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz"] Oct 31 01:40:16.770794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:16.770771 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e7ff10a-59849775dc-7r2mz"] Oct 31 01:40:17.042223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:17.042126 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:17.663910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:17.663878 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:17.745730 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:17.745690 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67ede1c1-bb24-450b-8dc3-0e56c847c3d0" path="/var/lib/kubelet/pods/67ede1c1-bb24-450b-8dc3-0e56c847c3d0/volumes" Oct 31 01:40:18.145085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:18.145045 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:18.444921 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:18.444810 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:18.966744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:18.966688 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:19.154079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:19.154033 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:19.299801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:19.299771 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:19.459082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:19.458960 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:19.624153 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:19.624064 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:19.874619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:19.874528 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:20.419913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:20.419874 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:20.565610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:20.565583 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:20.703902 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:20.703825 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:21.305932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:21.305897 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:21.447742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:21.447689 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:21.599339 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:21.599259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:21.739480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:21.739446 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:22.244202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.244165 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5"] Oct 31 01:40:22.244741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.244659 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" podUID="c17ea991-26db-4514-8f98-d658c1f4ec3b" containerName="maas-api" containerID="cri-o://bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0" gracePeriod=30 Oct 31 01:40:22.324753 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.324692 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:22.389739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.389544 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6"] Oct 31 01:40:22.389901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.389789 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" podUID="cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" containerName="maas-api" containerID="cri-o://0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82" gracePeriod=30 Oct 31 01:40:22.480933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.480893 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:22.603848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.603505 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" podUID="cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": dial tcp 10.132.0.59:8443: connect: connection refused" Oct 31 01:40:22.616383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.616354 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:22.648876 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.648854 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:40:22.714944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.714912 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:40:22.727858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.727833 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-metrics-certs\") pod \"c17ea991-26db-4514-8f98-d658c1f4ec3b\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " Oct 31 01:40:22.728047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.727949 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-maas-api-tls\") pod \"c17ea991-26db-4514-8f98-d658c1f4ec3b\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " Oct 31 01:40:22.728047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.727987 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17ea991-26db-4514-8f98-d658c1f4ec3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ea991-26db-4514-8f98-d658c1f4ec3b-openshift-service-ca\") pod \"c17ea991-26db-4514-8f98-d658c1f4ec3b\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " Oct 31 01:40:22.728180 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.728046 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17ea991-26db-4514-8f98-d658c1f4ec3b-kube-api-access-tz8kk\" (UniqueName: \"kubernetes.io/projected/c17ea991-26db-4514-8f98-d658c1f4ec3b-kube-api-access-tz8kk\") pod \"c17ea991-26db-4514-8f98-d658c1f4ec3b\" (UID: \"c17ea991-26db-4514-8f98-d658c1f4ec3b\") " Oct 31 01:40:22.728967 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.728674 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17ea991-26db-4514-8f98-d658c1f4ec3b-openshift-service-ca" pod "c17ea991-26db-4514-8f98-d658c1f4ec3b" (UID: "c17ea991-26db-4514-8f98-d658c1f4ec3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:22.731310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.731268 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17ea991-26db-4514-8f98-d658c1f4ec3b-kube-api-access-tz8kk" pod "c17ea991-26db-4514-8f98-d658c1f4ec3b" (UID: "c17ea991-26db-4514-8f98-d658c1f4ec3b"). InnerVolumeSpecName "kube-api-access-tz8kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:22.731613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.731583 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-maas-api-tls" pod "c17ea991-26db-4514-8f98-d658c1f4ec3b" (UID: "c17ea991-26db-4514-8f98-d658c1f4ec3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:22.731932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.731905 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-metrics-certs" pod "c17ea991-26db-4514-8f98-d658c1f4ec3b" (UID: "c17ea991-26db-4514-8f98-d658c1f4ec3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:22.761035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.760998 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:22.781388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.781304 5090 generic.go:356] "Generic (PLEG): container finished" podID="cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" containerID="0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82" exitCode=0 Oct 31 01:40:22.781564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.781431 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" event={"ID":"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a","Type":"ContainerDied","Data":"0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82"} Oct 31 01:40:22.781564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.781483 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" event={"ID":"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a","Type":"ContainerDied","Data":"13c015306496dbe4c81474ca5c859f0367eee0848d04d4b113a76f2dab09f958"} Oct 31 01:40:22.781564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.781406 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6" Oct 31 01:40:22.781564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.781513 5090 scope.go:122] "RemoveContainer" containerID="0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82" Oct 31 01:40:22.782976 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.782954 5090 generic.go:356] "Generic (PLEG): container finished" podID="c17ea991-26db-4514-8f98-d658c1f4ec3b" containerID="bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0" exitCode=0 Oct 31 01:40:22.783093 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.783008 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" Oct 31 01:40:22.783093 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.783063 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" event={"ID":"c17ea991-26db-4514-8f98-d658c1f4ec3b","Type":"ContainerDied","Data":"bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0"} Oct 31 01:40:22.783202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.783117 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5" event={"ID":"c17ea991-26db-4514-8f98-d658c1f4ec3b","Type":"ContainerDied","Data":"00cd8884699725fc2f8fbc4617c012b866d47e91a49be51715d7f97cfaa08503"} Oct 31 01:40:22.796094 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.796068 5090 scope.go:122] "RemoveContainer" containerID="0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82" Oct 31 01:40:22.796916 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:22.796889 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82\": container with ID starting with 0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82 not found: ID does not exist" containerID="0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82" Oct 31 01:40:22.797011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.796934 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82"} err="failed to get container status \"0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82\": rpc error: code = NotFound desc = could not find container \"0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82\": container with ID starting with 0307ade385910d9d78a4c78f3256ae477505ef328892c9fdc1c2e6ed355a3e82 not found: ID does not exist" Oct 31 01:40:22.797011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.796950 5090 scope.go:122] "RemoveContainer" containerID="bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0" Oct 31 01:40:22.807509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.807487 5090 scope.go:122] "RemoveContainer" containerID="bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0" Oct 31 01:40:22.807850 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:22.807828 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0\": container with ID starting with bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0 not found: ID does not exist" containerID="bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0" Oct 31 01:40:22.807935 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.807856 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0"} err="failed to get container status \"bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0\": rpc error: code = NotFound desc = could not find container \"bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0\": container with ID starting with bd2a4e17743768bdd7818952a52ac381d89594030fba9baf7577cfa5ccf439c0 not found: ID does not exist" Oct 31 01:40:22.816332 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.816307 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5"] Oct 31 01:40:22.823361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.823341 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4db52a49-64c6f567bc-qlkk5"] Oct 31 01:40:22.828999 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.828965 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-metrics-certs\") pod \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " Oct 31 01:40:22.829106 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.829021 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-openshift-service-ca\") pod \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " Oct 31 01:40:22.829106 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.829061 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-kube-api-access-l2twv\" (UniqueName: \"kubernetes.io/projected/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-kube-api-access-l2twv\") pod \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " Oct 31 01:40:22.829209 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.829161 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-maas-api-tls\") pod \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\" (UID: \"cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a\") " Oct 31 01:40:22.829793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.829497 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:22.829793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.829524 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17ea991-26db-4514-8f98-d658c1f4ec3b-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:22.829793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.829541 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz8kk\" (UniqueName: \"kubernetes.io/projected/c17ea991-26db-4514-8f98-d658c1f4ec3b-kube-api-access-tz8kk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:22.829793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.829559 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17ea991-26db-4514-8f98-d658c1f4ec3b-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:22.830226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.830195 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-openshift-service-ca" pod "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" (UID: "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:22.833935 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.833913 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-maas-api-tls" pod "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" (UID: "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:22.834336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.834303 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-metrics-certs" pod "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" (UID: "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:22.836462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.836384 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-kube-api-access-l2twv" pod "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" (UID: "cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a"). InnerVolumeSpecName "kube-api-access-l2twv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:22.913055 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.913023 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:22.930598 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.930570 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:22.930778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.930600 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:22.930778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.930617 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2twv\" (UniqueName: \"kubernetes.io/projected/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-kube-api-access-l2twv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:22.930778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:22.930631 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:23.078474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.078379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:23.105349 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.105318 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6"] Oct 31 01:40:23.112260 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.112237 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-aba34286-85464cd89-dvgp6"] Oct 31 01:40:23.227216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.227182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:23.367219 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.367141 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:23.702469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.702392 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:23.711270 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.711245 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9"] Oct 31 01:40:23.711468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.711445 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" podUID="5a45ee50-e1ea-425a-8795-9fba0c61c738" containerName="maas-api" containerID="cri-o://1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426" gracePeriod=30 Oct 31 01:40:23.750306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.750273 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17ea991-26db-4514-8f98-d658c1f4ec3b" path="/var/lib/kubelet/pods/c17ea991-26db-4514-8f98-d658c1f4ec3b/volumes" Oct 31 01:40:23.750976 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.750952 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" path="/var/lib/kubelet/pods/cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a/volumes" Oct 31 01:40:23.841195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.841166 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:23.991049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:23.991022 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:40:24.040778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.040709 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-metrics-certs\") pod \"5a45ee50-e1ea-425a-8795-9fba0c61c738\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " Oct 31 01:40:24.040933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.040817 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a45ee50-e1ea-425a-8795-9fba0c61c738-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a45ee50-e1ea-425a-8795-9fba0c61c738-openshift-service-ca\") pod \"5a45ee50-e1ea-425a-8795-9fba0c61c738\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " Oct 31 01:40:24.040933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.040869 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a45ee50-e1ea-425a-8795-9fba0c61c738-kube-api-access-566hl\" (UniqueName: \"kubernetes.io/projected/5a45ee50-e1ea-425a-8795-9fba0c61c738-kube-api-access-566hl\") pod \"5a45ee50-e1ea-425a-8795-9fba0c61c738\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " Oct 31 01:40:24.040933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.040907 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-maas-api-tls\") pod \"5a45ee50-e1ea-425a-8795-9fba0c61c738\" (UID: \"5a45ee50-e1ea-425a-8795-9fba0c61c738\") " Oct 31 01:40:24.041235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.041209 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a45ee50-e1ea-425a-8795-9fba0c61c738-openshift-service-ca" pod "5a45ee50-e1ea-425a-8795-9fba0c61c738" (UID: "5a45ee50-e1ea-425a-8795-9fba0c61c738"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:40:24.043018 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.042990 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-metrics-certs" pod "5a45ee50-e1ea-425a-8795-9fba0c61c738" (UID: "5a45ee50-e1ea-425a-8795-9fba0c61c738"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:24.043648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.043612 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-maas-api-tls" pod "5a45ee50-e1ea-425a-8795-9fba0c61c738" (UID: "5a45ee50-e1ea-425a-8795-9fba0c61c738"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:40:24.043648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.043645 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a45ee50-e1ea-425a-8795-9fba0c61c738-kube-api-access-566hl" pod "5a45ee50-e1ea-425a-8795-9fba0c61c738" (UID: "5a45ee50-e1ea-425a-8795-9fba0c61c738"). InnerVolumeSpecName "kube-api-access-566hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:40:24.048657 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.048590 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:24.142170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.142133 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:24.142338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.142174 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a45ee50-e1ea-425a-8795-9fba0c61c738-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:24.142338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.142194 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-566hl\" (UniqueName: \"kubernetes.io/projected/5a45ee50-e1ea-425a-8795-9fba0c61c738-kube-api-access-566hl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:24.142338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.142209 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a45ee50-e1ea-425a-8795-9fba0c61c738-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:40:24.273119 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.273087 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:24.423454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.423406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:24.566385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.566279 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:24.710289 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.710229 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:24.818571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.818492 5090 generic.go:356] "Generic (PLEG): container finished" podID="5a45ee50-e1ea-425a-8795-9fba0c61c738" containerID="1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426" exitCode=0 Oct 31 01:40:24.818741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.818675 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" event={"ID":"5a45ee50-e1ea-425a-8795-9fba0c61c738","Type":"ContainerDied","Data":"1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426"} Oct 31 01:40:24.818826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.818747 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" event={"ID":"5a45ee50-e1ea-425a-8795-9fba0c61c738","Type":"ContainerDied","Data":"15ba3c72c88bec88846a9b0edb5523768d143d0f584a3213dd0112507f2ea3c3"} Oct 31 01:40:24.818826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.818779 5090 scope.go:122] "RemoveContainer" containerID="1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426" Oct 31 01:40:24.818934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.818924 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9" Oct 31 01:40:24.835432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.835283 5090 scope.go:122] "RemoveContainer" containerID="1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426" Oct 31 01:40:24.835940 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:24.835902 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426\": container with ID starting with 1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426 not found: ID does not exist" containerID="1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426" Oct 31 01:40:24.836040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.835950 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426"} err="failed to get container status \"1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426\": rpc error: code = NotFound desc = could not find container \"1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426\": container with ID starting with 1a3d8dd4dfaac77f017fc1d111a414e5c94a514e6c191e9888d36861c494c426 not found: ID does not exist" Oct 31 01:40:24.845142 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.845119 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9"] Oct 31 01:40:24.853073 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.853049 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6198e156-68dfdb8584-kstb9"] Oct 31 01:40:24.908240 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.908211 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:24.971526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.971493 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs"] Oct 31 01:40:24.972111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972093 5090 cpu_manager.go:414] "containerMap: removing container" podUID="c17ea991-26db-4514-8f98-d658c1f4ec3b" containerName="maas-api" Oct 31 01:40:24.972196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972113 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17ea991-26db-4514-8f98-d658c1f4ec3b" containerName="maas-api" Oct 31 01:40:24.972196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972145 5090 cpu_manager.go:414] "containerMap: removing container" podUID="cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" containerName="maas-api" Oct 31 01:40:24.972196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972152 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" containerName="maas-api" Oct 31 01:40:24.972196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972188 5090 cpu_manager.go:414] "containerMap: removing container" podUID="5a45ee50-e1ea-425a-8795-9fba0c61c738" containerName="maas-api" Oct 31 01:40:24.972329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972197 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a45ee50-e1ea-425a-8795-9fba0c61c738" containerName="maas-api" Oct 31 01:40:24.972329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972211 5090 cpu_manager.go:414] "containerMap: removing container" podUID="67ede1c1-bb24-450b-8dc3-0e56c847c3d0" containerName="maas-api" Oct 31 01:40:24.972329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972218 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67ede1c1-bb24-450b-8dc3-0e56c847c3d0" containerName="maas-api" Oct 31 01:40:24.972329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972301 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17ea991-26db-4514-8f98-d658c1f4ec3b" containerName="maas-api" Oct 31 01:40:24.972329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972316 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe288d0-ef2a-4b1a-95eb-fc1c47bacb3a" containerName="maas-api" Oct 31 01:40:24.972484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972330 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a45ee50-e1ea-425a-8795-9fba0c61c738" containerName="maas-api" Oct 31 01:40:24.972484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.972340 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="67ede1c1-bb24-450b-8dc3-0e56c847c3d0" containerName="maas-api" Oct 31 01:40:24.977419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.977397 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:24.985544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.985516 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-be6935-serving-cert\"" Oct 31 01:40:24.990068 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:24.990047 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs"] Oct 31 01:40:25.052849 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.052822 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhhx\" (UniqueName: \"kubernetes.io/projected/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-kube-api-access-xjhhx\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.053020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.052884 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.053096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.053053 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.053148 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.053099 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.062472 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.062413 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:25.154136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.154062 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.154136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.154103 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.154399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.154243 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhhx\" (UniqueName: \"kubernetes.io/projected/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-kube-api-access-xjhhx\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.154399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.154311 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.154981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.154955 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.156676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.156657 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-metrics-certs\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.156808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.156757 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-maas-api-tls\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.166014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.165992 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhhx\" (UniqueName: \"kubernetes.io/projected/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-kube-api-access-xjhhx\") pod \"maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.210694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.210670 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:25.298509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.298470 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:25.371427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.371389 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:25.579592 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.579544 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs"] Oct 31 01:40:25.582361 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:40:25.582335 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb91b77_ea0e_4a2d_9402_41e876ff6f9e.slice/crio-031e6a9c6c2eefd3bc787ffc0168364f6a3d5aabd5c14e66935d7cae7769553c WatchSource:0}: Error finding container 031e6a9c6c2eefd3bc787ffc0168364f6a3d5aabd5c14e66935d7cae7769553c: Status 404 returned error can't find the container with id 031e6a9c6c2eefd3bc787ffc0168364f6a3d5aabd5c14e66935d7cae7769553c Oct 31 01:40:25.746850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.746817 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a45ee50-e1ea-425a-8795-9fba0c61c738" path="/var/lib/kubelet/pods/5a45ee50-e1ea-425a-8795-9fba0c61c738/volumes" Oct 31 01:40:25.796862 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.796826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:25.825410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.825380 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" event={"ID":"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e","Type":"ContainerStarted","Data":"031e6a9c6c2eefd3bc787ffc0168364f6a3d5aabd5c14e66935d7cae7769553c"} Oct 31 01:40:25.947249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:25.947177 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:26.259439 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:26.259398 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:26.402281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:26.402252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:26.834055 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:26.834016 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" event={"ID":"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e","Type":"ContainerStarted","Data":"e5ce57c17c029fb8465f2f507e45caa1583ed90d6a58af7947b054b24845b74f"} Oct 31 01:40:26.834459 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:26.834215 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:26.853314 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:26.853259 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" podStartSLOduration=2.406475655 podStartE2EDuration="2.853238318s" totalImagesPullingTime="446.762663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:26.852068534 +0000 UTC m=+1227.232313780" watchObservedRunningTime="2026-10-31 01:40:26.853238318 +0000 UTC m=+1227.233483542" Oct 31 01:40:26.924964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:26.924910 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:27.064097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:27.064063 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:27.306319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:27.306289 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:27.628455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:27.628358 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:27.774597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:27.774565 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:27.965146 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:27.965043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:28.311885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:28.311839 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:28.454101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:28.454068 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:28.620260 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:28.620182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:28.769196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:28.769161 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:29.089667 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:29.089625 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:29.237127 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:29.237088 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:29.386934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:29.386859 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:29.529390 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:29.529348 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:29.664605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:29.664519 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:30.019545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.019512 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:30.166821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.166790 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:30.674431 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.674394 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6"] Oct 31 01:40:30.678977 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.678954 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.687247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.687216 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-7e5955-serving-cert\"" Oct 31 01:40:30.693208 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.693180 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6"] Oct 31 01:40:30.738483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.738453 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:30.814852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.814819 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.815013 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.814995 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-metrics-certs\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.815105 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.815032 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6r4\" (UniqueName: \"kubernetes.io/projected/b3a029da-d748-484f-a127-645719398a9c-kube-api-access-cd6r4\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.815105 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.815085 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a029da-d748-484f-a127-645719398a9c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.894132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.894093 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:30.915627 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.915599 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-metrics-certs\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.915840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.915634 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6r4\" (UniqueName: \"kubernetes.io/projected/b3a029da-d748-484f-a127-645719398a9c-kube-api-access-cd6r4\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.915840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.915664 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a029da-d748-484f-a127-645719398a9c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.915840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.915769 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.916360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.916333 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a029da-d748-484f-a127-645719398a9c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.918215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.918188 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-metrics-certs\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.918311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.918266 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:30.927592 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:30.927546 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6r4\" (UniqueName: \"kubernetes.io/projected/b3a029da-d748-484f-a127-645719398a9c-kube-api-access-cd6r4\") pod \"maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:31.001728 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.001691 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:31.065703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.065675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:31.280936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.280908 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6"] Oct 31 01:40:31.284790 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:40:31.284755 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a029da_d748_484f_a127_645719398a9c.slice/crio-809f0348f279217f3a2b8b218dc0361f4234894fd43e7055df16919ae7ea875d WatchSource:0}: Error finding container 809f0348f279217f3a2b8b218dc0361f4234894fd43e7055df16919ae7ea875d: Status 404 returned error can't find the container with id 809f0348f279217f3a2b8b218dc0361f4234894fd43e7055df16919ae7ea875d Oct 31 01:40:31.662673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.662573 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:31.868419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.868366 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" event={"ID":"b3a029da-d748-484f-a127-645719398a9c","Type":"ContainerStarted","Data":"057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585"} Oct 31 01:40:31.868600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.868435 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" event={"ID":"b3a029da-d748-484f-a127-645719398a9c","Type":"ContainerStarted","Data":"809f0348f279217f3a2b8b218dc0361f4234894fd43e7055df16919ae7ea875d"} Oct 31 01:40:31.868600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.868465 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:31.887076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:31.887030 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" podStartSLOduration=1.39618109 podStartE2EDuration="1.887015877s" totalImagesPullingTime="490.834787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:31.886154425 +0000 UTC m=+1232.266399649" watchObservedRunningTime="2026-10-31 01:40:31.887015877 +0000 UTC m=+1232.267261097" Oct 31 01:40:32.244602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.244559 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x"] Oct 31 01:40:32.249139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.249120 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.259863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.259839 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-6ae519-serving-cert\"" Oct 31 01:40:32.262228 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.262204 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x"] Oct 31 01:40:32.329052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.329006 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef7beaa-db5f-408c-bd7a-b74a74248d63-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.329436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.329187 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-metrics-certs\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.329436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.329222 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2mwg\" (UniqueName: \"kubernetes.io/projected/eef7beaa-db5f-408c-bd7a-b74a74248d63-kube-api-access-q2mwg\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.329436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.329338 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.337144 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.337120 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:32.429978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.429950 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-metrics-certs\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.430167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.429989 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2mwg\" (UniqueName: \"kubernetes.io/projected/eef7beaa-db5f-408c-bd7a-b74a74248d63-kube-api-access-q2mwg\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.430167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.430122 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.430282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.430176 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef7beaa-db5f-408c-bd7a-b74a74248d63-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.430822 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.430796 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef7beaa-db5f-408c-bd7a-b74a74248d63-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.432562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.432538 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-maas-api-tls\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.432688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.432667 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-metrics-certs\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.441635 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.441613 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2mwg\" (UniqueName: \"kubernetes.io/projected/eef7beaa-db5f-408c-bd7a-b74a74248d63-kube-api-access-q2mwg\") pod \"maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.474348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.474318 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:32.574751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.574638 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:32.612544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.612511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:32.846413 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.846346 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 01:40:32.860537 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.860470 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x"] Oct 31 01:40:32.862869 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:40:32.862840 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef7beaa_db5f_408c_bd7a_b74a74248d63.slice/crio-24f7924d50ebdf48d815a7575fc48b0636b265d6790ecbd43aba121c3368168c WatchSource:0}: Error finding container 24f7924d50ebdf48d815a7575fc48b0636b265d6790ecbd43aba121c3368168c: Status 404 returned error can't find the container with id 24f7924d50ebdf48d815a7575fc48b0636b265d6790ecbd43aba121c3368168c Oct 31 01:40:32.889300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:32.889263 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" event={"ID":"eef7beaa-db5f-408c-bd7a-b74a74248d63","Type":"ContainerStarted","Data":"24f7924d50ebdf48d815a7575fc48b0636b265d6790ecbd43aba121c3368168c"} Oct 31 01:40:33.134222 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.134139 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx"] Oct 31 01:40:33.139836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.139809 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.148143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.148097 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-3cf6f4-serving-cert\"" Oct 31 01:40:33.153435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.153406 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx"] Oct 31 01:40:33.242085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.242051 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-metrics-certs\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.242251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.242122 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.242251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.242190 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5427-486c-496c-b778-518435b1e0d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.242251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.242244 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l26f\" (UniqueName: \"kubernetes.io/projected/56fa5427-486c-496c-b778-518435b1e0d3-kube-api-access-6l26f\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.258036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.257995 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:33.343845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.343810 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-metrics-certs\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.344234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.343881 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.344234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.343909 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5427-486c-496c-b778-518435b1e0d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.344234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.343932 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l26f\" (UniqueName: \"kubernetes.io/projected/56fa5427-486c-496c-b778-518435b1e0d3-kube-api-access-6l26f\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.344234 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:33.344041 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-serving-cert: secret "maas-api-e2e-worker-w0-3cf6f4-serving-cert" not found Oct 31 01:40:33.344234 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:33.344117 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls podName:56fa5427-486c-496c-b778-518435b1e0d3 nodeName:}" failed. No retries permitted until 2026-10-31 01:40:33.844092751 +0000 UTC m=+1234.224337958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls") pod "maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" (UID: "56fa5427-486c-496c-b778-518435b1e0d3") : secret "maas-api-e2e-worker-w0-3cf6f4-serving-cert" not found Oct 31 01:40:33.344647 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.344630 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5427-486c-496c-b778-518435b1e0d3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.346603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.346581 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-metrics-certs\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.358484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.358449 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l26f\" (UniqueName: \"kubernetes.io/projected/56fa5427-486c-496c-b778-518435b1e0d3-kube-api-access-6l26f\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.396195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.396169 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:33.532937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.532900 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:33.848470 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.848430 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.851214 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.851185 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:33.896081 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.895995 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" event={"ID":"eef7beaa-db5f-408c-bd7a-b74a74248d63","Type":"ContainerStarted","Data":"850f8a536e1d17e7aa1ab32424d015ed57abd2204fd62ea2bb8e4e308c506bd5"} Oct 31 01:40:33.896261 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.896143 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:33.915871 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:33.915825 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" podStartSLOduration=1.400355948 podStartE2EDuration="1.915810098s" totalImagesPullingTime="515.45415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:33.91362121 +0000 UTC m=+1234.293866444" watchObservedRunningTime="2026-10-31 01:40:33.915810098 +0000 UTC m=+1234.296055320" Oct 31 01:40:34.052848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.052815 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:34.061955 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.061923 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:34.339600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.339572 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx"] Oct 31 01:40:34.342248 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:40:34.342221 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fa5427_486c_496c_b778_518435b1e0d3.slice/crio-b3ecc7fd6a366a4f71f0cf736974a8a2e7af8dd4a3f1da290f250f2c91129e15 WatchSource:0}: Error finding container b3ecc7fd6a366a4f71f0cf736974a8a2e7af8dd4a3f1da290f250f2c91129e15: Status 404 returned error can't find the container with id b3ecc7fd6a366a4f71f0cf736974a8a2e7af8dd4a3f1da290f250f2c91129e15 Oct 31 01:40:34.742177 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.742138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:34.788791 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:40:34.788760 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:40:34.903060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.903027 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" event={"ID":"56fa5427-486c-496c-b778-518435b1e0d3","Type":"ContainerStarted","Data":"7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854"} Oct 31 01:40:34.903198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.903085 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" event={"ID":"56fa5427-486c-496c-b778-518435b1e0d3","Type":"ContainerStarted","Data":"b3ecc7fd6a366a4f71f0cf736974a8a2e7af8dd4a3f1da290f250f2c91129e15"} Oct 31 01:40:34.903393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.903374 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:34.924654 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:34.924594 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" podStartSLOduration=1.427532453 podStartE2EDuration="1.924579753s" totalImagesPullingTime="497.0473ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:40:34.922450883 +0000 UTC m=+1235.302696117" watchObservedRunningTime="2026-10-31 01:40:34.924579753 +0000 UTC m=+1235.304824975" Oct 31 01:40:35.507351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:35.507296 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:36.248761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:36.248724 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:36.928378 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:36.928343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:37.744148 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:37.744113 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:37.894950 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:37.894919 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 01:40:38.387531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:38.387498 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:38.561709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:38.561673 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:39.207538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:39.207507 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:39.361463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:39.361433 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:39.536839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:39.536813 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:39.910054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:39.909969 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 01:40:40.076395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:40.076359 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:40.916425 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:40.916395 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 01:40:41.019308 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:41.019276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:41.540877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:41.540844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:41.690629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:41.690591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:41.835491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:41.835404 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:42.578398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:42.578361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:42.718423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:42.718387 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:43.272235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:43.272199 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:43.416377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:43.416349 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:43.594240 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:43.594166 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:43.740696 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:43.740670 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:44.328895 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:44.328862 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:44.478990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:44.478957 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:44.634473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:44.634394 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:45.142684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:45.142645 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:45.300352 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:45.300307 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:46.070422 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:46.070391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:46.845443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:46.845391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:47.212288 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:47.212193 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:47.476602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:47.476513 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:47.637116 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:47.637079 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:47.867480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:47.867446 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:48.009300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:48.009270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:48.154068 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:48.153985 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:48.367678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:48.367644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:49.142066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:49.141994 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:49.496885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:49.496856 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:49.834385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:49.834304 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:50.534161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:50.534123 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:51.819574 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:51.819537 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:51.959656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:51.959628 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:52.661847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:52.661805 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:52.804079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:52.804045 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:53.344634 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:53.344600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:54.042357 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:54.042324 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:54.734710 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:54.734679 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:55.503197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:55.503162 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:55.650613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:55.650574 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:56.423895 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:56.423863 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:57.259709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:57.259672 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:57.945526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:57.945456 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:58.561429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:58.561387 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:59.238250 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:59.236689 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:40:59.858672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:40:59.858643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:00.549641 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:00.549602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:01.340404 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:01.340344 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:01.974607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:01.974569 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:02.557870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:02.557828 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:02.699408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:02.699375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:03.342897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:03.342868 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:03.999296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:03.999261 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:04.657685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:04.657655 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:04.789105 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:41:04.789063 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:41:05.357569 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:05.357536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:06.028745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:06.028692 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:06.751837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:06.751805 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:07.446978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:07.446944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:08.138018 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:08.137973 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:08.775662 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:08.775630 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:09.070771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:09.070656 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:09.638528 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:09.638498 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:10.341515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:10.341480 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:11.047771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:11.047709 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:11.830584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:11.830543 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:12.134458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:12.134327 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:12.528523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:12.528475 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:13.148433 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:13.148401 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:13.826568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:13.826519 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:14.553545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:14.553492 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:15.229863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:15.229827 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:15.952279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:15.952248 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:16.631083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:16.631053 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:16.801624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:16.801593 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:17.446728 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:17.446691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:18.133949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:18.133900 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:18.757815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:18.757783 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:19.464163 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:19.464124 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:20.072530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:20.072500 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:20.751615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:20.751582 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:21.569751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.566984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:21.830271 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.830173 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr"] Oct 31 01:41:21.834673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.834651 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:21.841145 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.841120 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"openshift-service-ca.crt\"" Oct 31 01:41:21.844548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.844521 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr"] Oct 31 01:41:21.846165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.846138 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"kube-root-ca.crt\"" Oct 31 01:41:21.846286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.846238 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"default-dockercfg-wwmcf\"" Oct 31 01:41:21.847080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.847035 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6ae519\"/\"premium-w2-6ae519-kserve-self-signed-certs\"" Oct 31 01:41:21.914621 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.914566 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:21.921913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.921881 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f19ad8a-396e-4322-abf4-f88698c9893c-tls-certs\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:21.922062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.921931 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4zp\" (UniqueName: \"kubernetes.io/projected/5f19ad8a-396e-4322-abf4-f88698c9893c-kube-api-access-7j4zp\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:21.922062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.922031 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-home\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:21.922153 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.922099 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-model-cache\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:21.922217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.922196 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-dshm\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:21.922264 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:21.922239 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-tmp-dir\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023005 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-model-cache\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023075 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-dshm\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023119 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-tmp-dir\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023186 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f19ad8a-396e-4322-abf4-f88698c9893c-tls-certs\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023217 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4zp\" (UniqueName: \"kubernetes.io/projected/5f19ad8a-396e-4322-abf4-f88698c9893c-kube-api-access-7j4zp\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023257 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-home\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023484 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-model-cache\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023563 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-home\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.023696 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.023661 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-tmp-dir\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.025800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.025774 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-dshm\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.025884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.025815 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f19ad8a-396e-4322-abf4-f88698c9893c-tls-certs\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.035144 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.035121 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4zp\" (UniqueName: \"kubernetes.io/projected/5f19ad8a-396e-4322-abf4-f88698c9893c-kube-api-access-7j4zp\") pod \"premium-w2-6ae519-kserve-847fd96895-h2rgr\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.155846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.155760 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:22.249611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.249570 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:22.530771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.530744 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr"] Oct 31 01:41:22.533909 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:41:22.533882 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f19ad8a_396e_4322_abf4_f88698c9893c.slice/crio-008f68d7425ce0dcc082d73c86980f79008ff4559a477c4888f469a014835e70 WatchSource:0}: Error finding container 008f68d7425ce0dcc082d73c86980f79008ff4559a477c4888f469a014835e70: Status 404 returned error can't find the container with id 008f68d7425ce0dcc082d73c86980f79008ff4559a477c4888f469a014835e70 Oct 31 01:41:22.535728 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:22.535703 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:41:23.139969 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:23.139889 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:23.181166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:23.181122 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" event={"ID":"5f19ad8a-396e-4322-abf4-f88698c9893c","Type":"ContainerStarted","Data":"008f68d7425ce0dcc082d73c86980f79008ff4559a477c4888f469a014835e70"} Oct 31 01:41:23.992400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:23.992360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:24.190941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:24.190907 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" event={"ID":"5f19ad8a-396e-4322-abf4-f88698c9893c","Type":"ContainerStarted","Data":"e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd"} Oct 31 01:41:24.210584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:24.210542 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" podStartSLOduration=1.7406865489999999 podStartE2EDuration="3.210528535s" totalImagesPullingTime="1.469841986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:41:24.208741946 +0000 UTC m=+1284.588987163" watchObservedRunningTime="2026-10-31 01:41:24.210528535 +0000 UTC m=+1284.590773758" Oct 31 01:41:24.671929 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:24.671900 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:25.156660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:25.156631 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:25.335573 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:25.335541 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:25.959320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:25.959285 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:26.603253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:26.603219 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:27.409253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:27.408970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:28.428940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:28.428907 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:29.150085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:29.150047 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:29.744931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:29.744904 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:30.440045 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:30.440017 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:31.134473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:31.134444 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:31.743023 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:31.742988 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:32.156824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:32.156751 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:32.169885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:32.169858 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:32.248973 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:32.248948 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 01:41:32.427102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:32.427021 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:33.040070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:33.040043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:33.659471 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:33.659441 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:34.349618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:34.349584 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:34.796338 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:41:34.796300 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:41:35.063615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:35.063528 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:35.926177 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:35.926147 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:36.742869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:36.742836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:37.624934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:37.624906 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:38.347426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:38.347389 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:39.147075 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:39.147030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:39.459661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:39.459568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:39.602407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:39.602376 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:39.822362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:39.822335 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:40.438458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:40.438416 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:41.138770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:41.138738 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:41.829842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:41.829812 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:42.589596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:42.589561 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:43.411322 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:43.411293 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:44.355867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:44.355835 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:45.203371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:45.203291 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:45.830632 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:45.830600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:46.533644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:46.533618 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:47.323870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:47.323831 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:47.950660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:47.950629 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:48.652240 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:48.652210 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:49.476322 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:49.476282 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:49.619841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:49.619804 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:50.175079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:50.175042 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:50.846027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:50.845992 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:51.575602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:51.575574 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:52.245114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:52.245070 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:52.947021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:52.946939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:53.737403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:53.737377 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:54.448338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:54.448310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:55.128355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:55.128315 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:55.767356 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:55.767328 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:56.541461 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:56.541397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:57.269292 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:57.269255 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:57.938586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:57.938550 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:58.643370 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:58.643343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:41:59.334411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:41:59.334379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:00.113833 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:00.113803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:00.955877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:00.955847 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:01.640695 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:01.640657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:02.362639 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:02.362601 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:03.110541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:03.110508 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:03.818596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:03.818560 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:04.472961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:04.472931 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:04.789317 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:42:04.789285 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:42:05.162534 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:05.162453 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:05.836305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:05.836276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:06.450385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:06.450353 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:07.068916 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:07.068883 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:07.676175 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:07.676125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:08.310322 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:08.310291 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:08.937526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:08.937491 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:09.568898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:09.568862 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:10.174661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:10.174612 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:10.845424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:10.845398 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:11.495982 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:11.495952 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:12.261478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:12.261433 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:12.735843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:12.735766 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:12.912642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:12.912608 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:13.126022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:13.125991 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:13.385460 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:13.385381 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:13.710985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:13.710828 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:13.856601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:13.856564 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:14.252334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:14.252294 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:14.971562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:14.971529 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:15.696217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:15.696158 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:16.359672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:16.359640 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:17.057448 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:17.057413 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:17.750969 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:17.750942 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:18.713685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:18.713646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:18.851003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:18.850970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:19.426402 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.426370 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:19.547642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.547614 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2"] Oct 31 01:42:19.551359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.551341 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.554551 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.554531 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-be6935-openshift-default-dockercfg-vj9xz\"" Oct 31 01:42:19.565961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.565939 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2"] Oct 31 01:42:19.597693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.597662 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.598966 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.598944 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wj9f\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-kube-api-access-9wj9f\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.599222 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.599201 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.599541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.599522 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.599764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.599746 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.600017 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.599998 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb292c29-cded-40f7-b40a-086df28415db-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.600233 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.600216 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.600446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.600429 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb292c29-cded-40f7-b40a-086df28415db-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.600662 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.600644 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702219 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702278 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wj9f\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-kube-api-access-9wj9f\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702315 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702422 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702450 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702486 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb292c29-cded-40f7-b40a-086df28415db-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702516 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702546 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb292c29-cded-40f7-b40a-086df28415db-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.702622 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.702573 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.704514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.703438 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.704514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.703465 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.704514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.703762 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.704514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.704460 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb292c29-cded-40f7-b40a-086df28415db-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.705040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.705020 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.708993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.708949 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb292c29-cded-40f7-b40a-086df28415db-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.709330 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.709291 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.716162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.716118 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wj9f\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-kube-api-access-9wj9f\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.718864 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.718822 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:19.864824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:19.864783 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:20.226830 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:42:20.226734 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb292c29_cded_40f7_b40a_086df28415db.slice/crio-38e80d8b7b265589464f642ff7fe77ca6fe94081016c0d19d3c89a91932dbdd8 WatchSource:0}: Error finding container 38e80d8b7b265589464f642ff7fe77ca6fe94081016c0d19d3c89a91932dbdd8: Status 404 returned error can't find the container with id 38e80d8b7b265589464f642ff7fe77ca6fe94081016c0d19d3c89a91932dbdd8 Oct 31 01:42:20.232216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.232191 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2"] Oct 31 01:42:20.234527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.234490 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:20.234640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.234572 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:20.234640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.234616 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:20.479530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.479288 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:20.535168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.534107 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" event={"ID":"bb292c29-cded-40f7-b40a-086df28415db","Type":"ContainerStarted","Data":"2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b"} Oct 31 01:42:20.535168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.534211 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" event={"ID":"bb292c29-cded-40f7-b40a-086df28415db","Type":"ContainerStarted","Data":"38e80d8b7b265589464f642ff7fe77ca6fe94081016c0d19d3c89a91932dbdd8"} Oct 31 01:42:20.561598 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.561546 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podStartSLOduration=1.561533665 podStartE2EDuration="1.561533665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:20.559824663 +0000 UTC m=+1340.940069885" watchObservedRunningTime="2026-10-31 01:42:20.561533665 +0000 UTC m=+1340.941778887" Oct 31 01:42:20.865436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:20.865358 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:21.172017 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:21.171916 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:21.867022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:21.866981 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:42:21.867492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:21.867048 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 01:42:21.872156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:21.872131 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:22.699152 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:22.699114 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:22.865846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:22.865812 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:42:22.866022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:22.865858 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 01:42:23.371884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:23.371854 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:23.865682 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:23.865653 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:42:23.865898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:23.865749 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 01:42:24.246551 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.246523 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:24.536291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.536211 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn"] Oct 31 01:42:24.540500 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.540480 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.544290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.544272 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7e5955-openshift-default-dockercfg-t99cw\"" Oct 31 01:42:24.553645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.553621 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn"] Oct 31 01:42:24.699447 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699408 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699447 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699455 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004317f-26ec-4a67-af3a-4801711d090d-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699488 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004317f-26ec-4a67-af3a-4801711d090d-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699544 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fj8\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-kube-api-access-k2fj8\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699599 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699750 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699791 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699897 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.699944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.699928 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801260 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801334 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801374 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801418 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801456 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801503 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801541 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004317f-26ec-4a67-af3a-4801711d090d-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801581 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004317f-26ec-4a67-af3a-4801711d090d-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.801607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.801606 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fj8\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-kube-api-access-k2fj8\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.802221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.802187 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.802627 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.802597 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004317f-26ec-4a67-af3a-4801711d090d-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.802931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.802660 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.803002 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.802928 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.803113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.803086 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.804281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.804250 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.804658 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.804637 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004317f-26ec-4a67-af3a-4801711d090d-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.813328 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.813309 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.813436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.813403 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fj8\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-kube-api-access-k2fj8\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.857426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.857401 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:24.866085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.866062 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:42:24.866163 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.866105 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 01:42:24.963588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:24.963524 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:25.221196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.221173 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn"] Oct 31 01:42:25.223098 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:42:25.223061 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf004317f_26ec_4a67_af3a_4801711d090d.slice/crio-0a260a71d9f241e46b7dbea911ef8225d101e49a3881312b01f1e3a073a199f3 WatchSource:0}: Error finding container 0a260a71d9f241e46b7dbea911ef8225d101e49a3881312b01f1e3a073a199f3: Status 404 returned error can't find the container with id 0a260a71d9f241e46b7dbea911ef8225d101e49a3881312b01f1e3a073a199f3 Oct 31 01:42:25.225579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.225540 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:25.225645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.225628 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:25.225689 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.225672 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:25.565386 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.565351 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" event={"ID":"f004317f-26ec-4a67-af3a-4801711d090d","Type":"ContainerStarted","Data":"9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79"} Oct 31 01:42:25.565896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.565410 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" event={"ID":"f004317f-26ec-4a67-af3a-4801711d090d","Type":"ContainerStarted","Data":"0a260a71d9f241e46b7dbea911ef8225d101e49a3881312b01f1e3a073a199f3"} Oct 31 01:42:25.600749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.598230 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podStartSLOduration=1.598211814 podStartE2EDuration="1.598211814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:25.588342939 +0000 UTC m=+1345.968588162" watchObservedRunningTime="2026-10-31 01:42:25.598211814 +0000 UTC m=+1345.978457038" Oct 31 01:42:25.846021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.845943 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:25.858087 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.858056 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:25.865594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.865568 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:42:25.865763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:25.865624 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 01:42:26.697056 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:26.692672 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:26.859432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:26.859392 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:42:26.859572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:26.859443 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 01:42:26.865509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:26.865489 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 01:42:26.865553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:26.865537 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 01:42:27.372873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:27.372840 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:27.858336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:27.858297 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:42:27.858763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:27.858367 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 01:42:27.870229 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:27.870199 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:27.870863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:27.870837 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:27.871668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:27.871644 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 01:42:28.263390 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:28.263361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:28.858399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:28.858351 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:42:28.858944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:28.858419 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 01:42:29.063732 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:29.063693 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:29.859081 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:29.859044 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:42:29.859637 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:29.859609 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 01:42:30.022527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.022486 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:30.708856 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.708805 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb"] Oct 31 01:42:30.715451 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.715426 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.722896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.722871 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"default-dockercfg-z5c5f\"" Oct 31 01:42:30.725820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.723286 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"kube-root-ca.crt\"" Oct 31 01:42:30.725978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.723320 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-tmp-dir\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.726143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.726125 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-dshm\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.726911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.726306 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-525kd\" (UniqueName: \"kubernetes.io/projected/428c4a53-4c61-448c-af74-bb431b2fad7f-kube-api-access-525kd\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.726911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.726355 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-model-cache\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.726911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.726387 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428c4a53-4c61-448c-af74-bb431b2fad7f-tls-certs\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.726911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.726532 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-home\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.726911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.723409 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"openshift-service-ca.crt\"" Oct 31 01:42:30.726911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.724101 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"e2e-conflict-a-36645eac-kserve-self-signed-certs\"" Oct 31 01:42:30.729412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.729378 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb"] Oct 31 01:42:30.807144 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.807113 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:30.827150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.827113 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-525kd\" (UniqueName: \"kubernetes.io/projected/428c4a53-4c61-448c-af74-bb431b2fad7f-kube-api-access-525kd\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.827312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.827159 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-model-cache\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.827312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.827181 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428c4a53-4c61-448c-af74-bb431b2fad7f-tls-certs\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.827312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.827269 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-home\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.827312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.827288 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-tmp-dir\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.827549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.827315 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-dshm\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.828099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.828065 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-home\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.828337 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.828312 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-model-cache\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.828447 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.828398 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-tmp-dir\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.830963 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.830946 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-dshm\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.831119 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.831096 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428c4a53-4c61-448c-af74-bb431b2fad7f-tls-certs\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.839511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.839485 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-525kd\" (UniqueName: \"kubernetes.io/projected/428c4a53-4c61-448c-af74-bb431b2fad7f-kube-api-access-525kd\") pod \"e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:30.858204 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.858167 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 31 01:42:30.858353 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.858221 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 31 01:42:30.991469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:30.991323 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:31.034096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:31.034063 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:31.345037 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:31.345014 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb"] Oct 31 01:42:31.347889 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:42:31.347852 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod428c4a53_4c61_448c_af74_bb431b2fad7f.slice/crio-d1931b9ecd96f0cd376c421d1f12ec317dc22acd0a29a5d997857e3943c6df84 WatchSource:0}: Error finding container d1931b9ecd96f0cd376c421d1f12ec317dc22acd0a29a5d997857e3943c6df84: Status 404 returned error can't find the container with id d1931b9ecd96f0cd376c421d1f12ec317dc22acd0a29a5d997857e3943c6df84 Oct 31 01:42:31.645926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:31.645837 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" event={"ID":"428c4a53-4c61-448c-af74-bb431b2fad7f","Type":"ContainerStarted","Data":"c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50"} Oct 31 01:42:31.645926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:31.645884 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" event={"ID":"428c4a53-4c61-448c-af74-bb431b2fad7f","Type":"ContainerStarted","Data":"d1931b9ecd96f0cd376c421d1f12ec317dc22acd0a29a5d997857e3943c6df84"} Oct 31 01:42:31.664509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:31.664461 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" podStartSLOduration=1.664445188 podStartE2EDuration="1.664445188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:31.663100253 +0000 UTC m=+1352.043345502" watchObservedRunningTime="2026-10-31 01:42:31.664445188 +0000 UTC m=+1352.044690411" Oct 31 01:42:31.762727 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:31.762651 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:32.034905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:32.034863 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:32.095128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:32.095084 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:32.095559 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:32.095535 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:32.096460 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:32.096439 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 01:42:32.442697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:32.442613 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:33.251346 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:33.251309 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:33.955618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:33.955582 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:34.662111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:34.662079 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:34.789275 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:42:34.789241 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:42:35.447558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:35.447526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:36.245736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:36.245673 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:37.095011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:37.094977 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:38.072086 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.072049 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:38.084754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.084704 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m"] Oct 31 01:42:38.094359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.094332 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.115041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.115002 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6ae519-openshift-default-dockercfg-kmlj9\"" Oct 31 01:42:38.119973 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.119938 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m"] Oct 31 01:42:38.177952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.177915 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d090668-ad32-413e-9768-360d58a3a7cc-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.178207 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.178186 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d090668-ad32-413e-9768-360d58a3a7cc-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.178348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.178334 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.178458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.178445 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.178560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.178541 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.178665 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.178652 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264n9\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-kube-api-access-264n9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.178798 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.178783 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.178933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.178919 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.179066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.179053 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.232778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.226446 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:38.285325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.283623 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.285325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.283686 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.285325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.283745 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.285325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.283788 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264n9\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-kube-api-access-264n9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.285325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.283828 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.285325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.284313 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.285325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.284944 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.286843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.286810 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.287082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.287061 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.310926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.310895 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d090668-ad32-413e-9768-360d58a3a7cc-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.332852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.332781 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d090668-ad32-413e-9768-360d58a3a7cc-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.333111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.319537 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.354795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.325185 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d090668-ad32-413e-9768-360d58a3a7cc-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.355038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.326985 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.355140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.329404 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264n9\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-kube-api-access-264n9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.355231 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.331984 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.355315 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.311440 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.355387 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.333566 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d090668-ad32-413e-9768-360d58a3a7cc-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.415367 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.415331 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:38.703172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.703092 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m"] Oct 31 01:42:38.706980 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:42:38.706950 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d090668_ad32_413e_9768_360d58a3a7cc.slice/crio-5cce4ed46734719434ac18a7741adad2ebfa37961be7c358e4ce66c72020900f WatchSource:0}: Error finding container 5cce4ed46734719434ac18a7741adad2ebfa37961be7c358e4ce66c72020900f: Status 404 returned error can't find the container with id 5cce4ed46734719434ac18a7741adad2ebfa37961be7c358e4ce66c72020900f Oct 31 01:42:38.709904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.709869 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:38.709996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.709943 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:38.709996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.709973 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:38.952453 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:38.952422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:39.456402 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.456371 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt"] Oct 31 01:42:39.460301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.460281 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.479240 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.477957 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0e8b5c-openshift-default-dockercfg-pq7tc\"" Oct 31 01:42:39.499741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.499379 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt"] Oct 31 01:42:39.546212 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546172 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546212 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546208 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546298 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546322 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcpv4\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-kube-api-access-gcpv4\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546345 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546430 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546474 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546490 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17cbd44-ea63-47ff-8af6-689574aaea89-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.546769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.546527 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647440 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647486 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647520 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647655 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcpv4\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-kube-api-access-gcpv4\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647707 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647797 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647877 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.647940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647900 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17cbd44-ea63-47ff-8af6-689574aaea89-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.648031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.647985 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.648454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.648425 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.648820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.648766 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.648820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.648793 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.649016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.648856 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.649562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.649533 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17cbd44-ea63-47ff-8af6-689574aaea89-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.650942 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.650914 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.651258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.651234 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.670858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.670835 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.670972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.670898 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcpv4\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-kube-api-access-gcpv4\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:39.736468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.736380 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" event={"ID":"0d090668-ad32-413e-9768-360d58a3a7cc","Type":"ContainerStarted","Data":"f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b"} Oct 31 01:42:39.736468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.736442 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" event={"ID":"0d090668-ad32-413e-9768-360d58a3a7cc","Type":"ContainerStarted","Data":"5cce4ed46734719434ac18a7741adad2ebfa37961be7c358e4ce66c72020900f"} Oct 31 01:42:39.744319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.744292 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:39.760137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.760094 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podStartSLOduration=1.760081686 podStartE2EDuration="1.760081686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:39.757258175 +0000 UTC m=+1360.137503413" watchObservedRunningTime="2026-10-31 01:42:39.760081686 +0000 UTC m=+1360.140326902" Oct 31 01:42:39.797761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:39.793930 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:40.209541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.194988 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt"] Oct 31 01:42:40.212520 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:42:40.209760 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17cbd44_ea63_47ff_8af6_689574aaea89.slice/crio-92df8b90e0bdcdf5be4c54ceacb9399abdc63b9478212efc6faa505366810127 WatchSource:0}: Error finding container 92df8b90e0bdcdf5be4c54ceacb9399abdc63b9478212efc6faa505366810127: Status 404 returned error can't find the container with id 92df8b90e0bdcdf5be4c54ceacb9399abdc63b9478212efc6faa505366810127 Oct 31 01:42:40.212520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.210119 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:40.212520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.212305 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:40.212520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.212390 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:40.212520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.212435 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:40.415471 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.415439 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:40.417147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.417125 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:42:40.417248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.417176 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 01:42:40.756426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.756382 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" event={"ID":"b17cbd44-ea63-47ff-8af6-689574aaea89","Type":"ContainerStarted","Data":"7b6fd1a7a5d30376549b5cc9c14b3319baeb617ebf4c767f1945c94d2930548d"} Oct 31 01:42:40.756884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.756454 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" event={"ID":"b17cbd44-ea63-47ff-8af6-689574aaea89","Type":"ContainerStarted","Data":"92df8b90e0bdcdf5be4c54ceacb9399abdc63b9478212efc6faa505366810127"} Oct 31 01:42:40.810743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.794978 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podStartSLOduration=1.794955452 podStartE2EDuration="1.794955452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:40.789179073 +0000 UTC m=+1361.169424310" watchObservedRunningTime="2026-10-31 01:42:40.794955452 +0000 UTC m=+1361.175200676" Oct 31 01:42:40.812140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:40.812110 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:41.034998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.034903 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:41.048698 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.048562 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:41.054908 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.054881 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:41.422803 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.422516 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:42:41.422803 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.422597 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 01:42:41.772638 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.772613 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:42:41.802741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.800359 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:42:41.802741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:41.800422 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 01:42:42.011751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:42.009958 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:42.419740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:42.416170 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:42:42.419740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:42.416233 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 01:42:42.796117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:42.796079 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:42:42.796746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:42.796686 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 01:42:42.888739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:42.885784 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:43.424787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:43.424587 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:42:43.424787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:43.424671 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 01:42:43.799742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:43.799245 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:42:43.799742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:43.799301 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 01:42:43.924439 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:43.924407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:44.426233 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:44.426184 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:42:44.426440 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:44.426249 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 01:42:44.809747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:44.805220 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:42:44.810345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:44.810309 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 01:42:44.812137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:44.812111 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:45.097226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.094771 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486"] Oct 31 01:42:45.116801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.112911 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486"] Oct 31 01:42:45.116801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.113031 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164017 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.163987 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164232 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164394 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164511 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164670 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164652 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7f8b\" (UniqueName: \"kubernetes.io/projected/9de91f6a-4968-47b5-85bb-58d455c37a07-kube-api-access-b7f8b\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164694 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164747 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164778 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de91f6a-4968-47b5-85bb-58d455c37a07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.164887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.164803 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266056 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266014 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7f8b\" (UniqueName: \"kubernetes.io/projected/9de91f6a-4968-47b5-85bb-58d455c37a07-kube-api-access-b7f8b\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266255 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266067 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266255 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266091 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266255 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266116 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de91f6a-4968-47b5-85bb-58d455c37a07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266255 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266141 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266255 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266213 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266278 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266395 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266424 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266559 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.266663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.266609 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.267048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.267014 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de91f6a-4968-47b5-85bb-58d455c37a07-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.267274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.267252 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.267368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.267349 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.269625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.269605 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.269889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.269871 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.294738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.294535 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de91f6a-4968-47b5-85bb-58d455c37a07-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.294738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.294684 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7f8b\" (UniqueName: \"kubernetes.io/projected/9de91f6a-4968-47b5-85bb-58d455c37a07-kube-api-access-b7f8b\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-ks486\" (UID: \"9de91f6a-4968-47b5-85bb-58d455c37a07\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.416268 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.416187 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 31 01:42:45.416268 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.416246 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 31 01:42:45.436319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.436291 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:45.784121 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.781142 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486"] Oct 31 01:42:45.803747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.800488 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:42:45.803747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.800552 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 01:42:45.824062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.823462 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:45.824644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.824570 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:45.824835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.824673 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:45.824835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.824753 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:45.829112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:45.829086 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" event={"ID":"9de91f6a-4968-47b5-85bb-58d455c37a07","Type":"ContainerStarted","Data":"758225f9a33a8af33be748f9d91badf9e855df33e686f3523a62f0de6815b717"} Oct 31 01:42:46.420523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.420492 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:46.420791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.420760 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:46.421626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.421606 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 01:42:46.619132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.619020 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:46.801770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.800939 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 01:42:46.801770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.801005 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 01:42:46.836379 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.836342 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" event={"ID":"9de91f6a-4968-47b5-85bb-58d455c37a07","Type":"ContainerStarted","Data":"7aa770da8f800857bc0413aa20a3272bbde0493e6e9addcdc06e909e18a10557"} Oct 31 01:42:46.860860 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:46.860805 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podStartSLOduration=1.8607913040000001 podStartE2EDuration="1.860791304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:46.858007183 +0000 UTC m=+1367.238252428" watchObservedRunningTime="2026-10-31 01:42:46.860791304 +0000 UTC m=+1367.241036524" Oct 31 01:42:47.445780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:47.443169 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:47.458745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:47.456034 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:42:47.458745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:47.456097 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 01:42:47.479743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:47.467689 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:47.808979 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:47.808948 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:47.810699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:47.810677 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:47.811893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:47.811874 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 01:42:48.008686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:48.008641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:48.438380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:48.438331 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:42:48.438588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:48.438416 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 01:42:49.042745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:49.041509 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:49.438566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:49.438072 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:42:49.438566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:49.438135 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 01:42:49.672874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:49.672825 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:50.045062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:50.045009 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:50.436688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:50.436605 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:42:50.436688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:50.436665 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 01:42:50.853561 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:50.853529 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:51.437234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:51.437203 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:42:51.437620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:51.437254 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 01:42:51.578491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:51.578460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:52.347707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:52.347676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:52.437044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:52.437005 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:42:52.437222 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:52.437055 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 01:42:53.020166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.020130 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:53.405239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.405155 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b"] Oct 31 01:42:53.412108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.412081 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.417351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.417311 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b"] Oct 31 01:42:53.423855 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.420923 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3cf6f4\"/\"e2e-conflict-b-36645eac-kserve-self-signed-certs\"" Oct 31 01:42:53.437466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.437441 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 01:42:53.437656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.437631 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 01:42:53.471846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.471814 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24b2\" (UniqueName: \"kubernetes.io/projected/3a552884-1952-4323-9dcc-355fb200b2a0-kube-api-access-f24b2\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.472231 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.472203 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-model-cache\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.472346 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.472282 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-home\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.472346 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.472334 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-tmp-dir\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.472458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.472388 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a552884-1952-4323-9dcc-355fb200b2a0-tls-certs\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.472458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.472447 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-dshm\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.492760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.492730 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.573643 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-dshm\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.573695 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24b2\" (UniqueName: \"kubernetes.io/projected/3a552884-1952-4323-9dcc-355fb200b2a0-kube-api-access-f24b2\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.573803 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-model-cache\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.573858 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-home\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.573881 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-tmp-dir\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.573926 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a552884-1952-4323-9dcc-355fb200b2a0-tls-certs\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.574624 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-model-cache\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.575740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.575555 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-home\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.576309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.575860 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-tmp-dir\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.589404 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.589373 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a552884-1952-4323-9dcc-355fb200b2a0-tls-certs\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.589896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.589869 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-dshm\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.592234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.592209 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24b2\" (UniqueName: \"kubernetes.io/projected/3a552884-1952-4323-9dcc-355fb200b2a0-kube-api-access-f24b2\") pod \"e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.732812 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.732700 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:53.931023 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:53.930987 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:54.024061 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:54.024038 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b"] Oct 31 01:42:54.026968 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:42:54.026924 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a552884_1952_4323_9dcc_355fb200b2a0.slice/crio-7dca026a799c2b4c90448a1fb6dff1656f9bafb3ee6b355f95a31f70b65d918b WatchSource:0}: Error finding container 7dca026a799c2b4c90448a1fb6dff1656f9bafb3ee6b355f95a31f70b65d918b: Status 404 returned error can't find the container with id 7dca026a799c2b4c90448a1fb6dff1656f9bafb3ee6b355f95a31f70b65d918b Oct 31 01:42:54.729093 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:54.729040 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:54.921668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:54.921635 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" event={"ID":"3a552884-1952-4323-9dcc-355fb200b2a0","Type":"ContainerStarted","Data":"2e9e4c1c9452cc36e0fa081f6dee85accb402856a94cf824702d27c4c01cb9d6"} Oct 31 01:42:54.921867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:54.921682 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" event={"ID":"3a552884-1952-4323-9dcc-355fb200b2a0","Type":"ContainerStarted","Data":"7dca026a799c2b4c90448a1fb6dff1656f9bafb3ee6b355f95a31f70b65d918b"} Oct 31 01:42:54.942846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:54.942800 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" podStartSLOduration=1.942788594 podStartE2EDuration="1.942788594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:54.939465503 +0000 UTC m=+1375.319710726" watchObservedRunningTime="2026-10-31 01:42:54.942788594 +0000 UTC m=+1375.323033817" Oct 31 01:42:55.437567 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:55.437530 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:42:55.438124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:55.438086 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:42:55.539735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:55.539686 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:55.733598 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:55.733497 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:42:56.408790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.408750 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc"] Oct 31 01:42:56.415427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.415402 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.431349 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.431308 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3cf6f4-openshift-default-dockercfg-5t5rj\"" Oct 31 01:42:56.434700 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.433210 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc"] Oct 31 01:42:56.479557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.478869 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-ks486 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:42:56.479557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.478927 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" podUID="9de91f6a-4968-47b5-85bb-58d455c37a07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523226 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523291 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523343 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523395 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523442 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523539 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523596 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523675 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.526904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.523815 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt468\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-kube-api-access-xt468\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.529554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.529373 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:56.627435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627388 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627452 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627509 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627587 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt468\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-kube-api-access-xt468\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627630 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627656 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627687 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627752 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.627837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.627785 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.628304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.628275 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.628600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.628573 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.628893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.628867 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.629171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.629143 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.629486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.629439 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.632795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.632123 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.632795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.632615 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.643286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.642603 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.643286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.643222 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt468\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-kube-api-access-xt468\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:56.749210 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:56.749179 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:57.050586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.050560 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc"] Oct 31 01:42:57.053120 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:42:57.053090 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb06cda_3663_461c_93ca_7f63f7c12aa1.slice/crio-68b490c144ec11331268f53e6690e58d6bd77e9a01e6510d8cd3957c0d74d50d WatchSource:0}: Error finding container 68b490c144ec11331268f53e6690e58d6bd77e9a01e6510d8cd3957c0d74d50d: Status 404 returned error can't find the container with id 68b490c144ec11331268f53e6690e58d6bd77e9a01e6510d8cd3957c0d74d50d Oct 31 01:42:57.056069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.056025 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:57.056176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.056104 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:57.056176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.056153 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:42:57.311371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.311284 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:57.311697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.311675 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:57.312452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.312429 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-ks486" Oct 31 01:42:57.364480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.364445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:57.950027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.949994 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" event={"ID":"fcb06cda-3663-461c-93ca-7f63f7c12aa1","Type":"ContainerStarted","Data":"2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d"} Oct 31 01:42:57.950473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.950052 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" event={"ID":"fcb06cda-3663-461c-93ca-7f63f7c12aa1","Type":"ContainerStarted","Data":"68b490c144ec11331268f53e6690e58d6bd77e9a01e6510d8cd3957c0d74d50d"} Oct 31 01:42:57.973135 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:57.973077 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podStartSLOduration=1.973058763 podStartE2EDuration="1.973058763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:42:57.971091477 +0000 UTC m=+1378.351336718" watchObservedRunningTime="2026-10-31 01:42:57.973058763 +0000 UTC m=+1378.353303987" Oct 31 01:42:58.062452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:58.062422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:58.749300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:58.749269 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:42:58.751311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:58.751286 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:42:58.751451 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:58.751348 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 01:42:58.830567 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:58.830532 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:59.658735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:59.658684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:42:59.750060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:59.750023 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:42:59.750239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:42:59.750087 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 01:43:00.351185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:00.351125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:00.705702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:00.705605 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:00.749918 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:00.749889 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:43:00.750076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:00.749942 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 01:43:01.455442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:01.455405 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:01.749774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:01.749748 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:43:01.750169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:01.749801 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 01:43:02.271215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:02.271159 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:02.750495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:02.750463 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:43:02.751024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:02.750553 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 01:43:02.964709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:02.964676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:03.713562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:03.713528 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:03.733089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:03.733055 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:43:03.749865 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:03.749834 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Oct 31 01:43:03.750030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:03.749895 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Oct 31 01:43:03.752620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:03.752596 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:43:03.851767 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:03.851686 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:04.036635 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:04.036606 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:43:04.559549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:04.559513 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:04.754689 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:04.754654 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:43:04.755183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:04.754932 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:43:04.755995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:04.755972 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 01:43:04.788793 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:43:04.788761 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:43:05.439152 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:05.439111 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:06.144326 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:06.144283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:06.865642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:06.865599 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:07.726536 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:07.726509 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:08.489165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:08.489134 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:09.499481 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:09.499451 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:09.507154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:09.506811 5090 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:09.507154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:09.506871 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9vpv6" podUID="a3621e1c-e276-4304-a11d-8a44a339ea04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:10.424375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:10.424278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:11.038941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:11.038908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:11.739482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:11.739424 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:12.341874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:12.341838 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:12.968683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:12.968636 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:13.576275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:13.576234 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:13.967652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:13.967567 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:14.317876 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:14.317843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:15.017393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:15.017352 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b"] Oct 31 01:43:15.020086 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:15.020057 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:15.169608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:15.169580 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:15.846556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:15.846516 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:16.625040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:16.624993 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:17.351162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:17.351127 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:18.110609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:18.110571 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:18.861880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:18.861843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:19.656974 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:19.656932 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:20.464140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:20.464104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:20.612562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:20.612527 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb"] Oct 31 01:43:20.691506 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:20.691471 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:21.136919 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:21.136881 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:21.955923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:21.955890 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:22.623296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:22.623259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:23.319844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:23.319810 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:24.315287 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.315237 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6"] Oct 31 01:43:24.320674 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.320638 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.329316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.329268 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vfdlj\"" Oct 31 01:43:24.368815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.334681 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 01:43:24.369038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.334754 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 01:43:24.369174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.334798 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 01:43:24.369291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.349876 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6"] Oct 31 01:43:24.447691 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.447660 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.447897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.447710 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pnq\" (UniqueName: \"kubernetes.io/projected/5c521031-f454-4d7a-a4d6-f460b8df6560-kube-api-access-52pnq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.447897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.447762 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.447897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.447800 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.447897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.447841 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.447897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.447884 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c521031-f454-4d7a-a4d6-f460b8df6560-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.448219 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.447921 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.549508 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c521031-f454-4d7a-a4d6-f460b8df6560-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.549650 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.549745 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.549807 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pnq\" (UniqueName: \"kubernetes.io/projected/5c521031-f454-4d7a-a4d6-f460b8df6560-kube-api-access-52pnq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.549847 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.549884 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.549933 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.551746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.551364 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.562029 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.561996 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.562209 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.562086 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.562209 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.562148 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c521031-f454-4d7a-a4d6-f460b8df6560-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.565025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.563677 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.565025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.564227 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c521031-f454-4d7a-a4d6-f460b8df6560-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.565025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.564921 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pnq\" (UniqueName: \"kubernetes.io/projected/5c521031-f454-4d7a-a4d6-f460b8df6560-kube-api-access-52pnq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ddrf6\" (UID: \"5c521031-f454-4d7a-a4d6-f460b8df6560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.671844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.671762 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:24.966539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:24.966460 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6"] Oct 31 01:43:24.974793 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:43:24.974699 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c521031_f454_4d7a_a4d6_f460b8df6560.slice/crio-04efe5cc5b4158c875933d2a6078e7d7cd9947ff541e356147c5b6a015355955 WatchSource:0}: Error finding container 04efe5cc5b4158c875933d2a6078e7d7cd9947ff541e356147c5b6a015355955: Status 404 returned error can't find the container with id 04efe5cc5b4158c875933d2a6078e7d7cd9947ff541e356147c5b6a015355955 Oct 31 01:43:25.163882 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:25.163847 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" event={"ID":"5c521031-f454-4d7a-a4d6-f460b8df6560","Type":"ContainerStarted","Data":"04efe5cc5b4158c875933d2a6078e7d7cd9947ff541e356147c5b6a015355955"} Oct 31 01:43:26.384524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:26.384487 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:26.542253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:26.542215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:26.679362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:26.679274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:26.892842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:26.892808 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:27.340214 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:27.340186 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:27.864168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:27.864136 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:27.864470 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:27.864206 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:28.069954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:28.069919 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:28.185846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:28.185754 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" event={"ID":"5c521031-f454-4d7a-a4d6-f460b8df6560","Type":"ContainerStarted","Data":"326b38f92161a45422010dad610e554f459cd30232c151b98f81a9a412aa3325"} Oct 31 01:43:28.185846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:28.185819 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:28.187914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:28.187889 5090 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-ddrf6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:43:28.188046 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:28.187946 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" podUID="5c521031-f454-4d7a-a4d6-f460b8df6560" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:43:28.210677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:28.210632 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" podStartSLOduration=1.326352667 podStartE2EDuration="4.210614836s" totalImagesPullingTime="2.884262169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:28.207853858 +0000 UTC m=+1408.588099106" watchObservedRunningTime="2026-10-31 01:43:28.210614836 +0000 UTC m=+1408.590860061" Oct 31 01:43:28.829122 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:28.829088 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:29.202965 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:29.202884 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ddrf6" Oct 31 01:43:29.544171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:29.544141 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:30.043309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.043254 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" podUID="3a552884-1952-4323-9dcc-355fb200b2a0" containerName="main" containerID="cri-o://2e9e4c1c9452cc36e0fa081f6dee85accb402856a94cf824702d27c4c01cb9d6" gracePeriod=45 Oct 31 01:43:30.166474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.166449 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:30.205275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.205232 5090 generic.go:356] "Generic (PLEG): container finished" podID="3a552884-1952-4323-9dcc-355fb200b2a0" containerID="2e9e4c1c9452cc36e0fa081f6dee85accb402856a94cf824702d27c4c01cb9d6" exitCode=0 Oct 31 01:43:30.205868 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.205543 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" event={"ID":"3a552884-1952-4323-9dcc-355fb200b2a0","Type":"ContainerDied","Data":"2e9e4c1c9452cc36e0fa081f6dee85accb402856a94cf824702d27c4c01cb9d6"} Oct 31 01:43:30.343640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.343532 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:43:30.424338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424298 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-model-cache\") pod \"3a552884-1952-4323-9dcc-355fb200b2a0\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " Oct 31 01:43:30.424503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424405 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-tmp-dir\") pod \"3a552884-1952-4323-9dcc-355fb200b2a0\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " Oct 31 01:43:30.424503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424476 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a552884-1952-4323-9dcc-355fb200b2a0-kube-api-access-f24b2\" (UniqueName: \"kubernetes.io/projected/3a552884-1952-4323-9dcc-355fb200b2a0-kube-api-access-f24b2\") pod \"3a552884-1952-4323-9dcc-355fb200b2a0\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " Oct 31 01:43:30.424617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424534 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-model-cache" pod "3a552884-1952-4323-9dcc-355fb200b2a0" (UID: "3a552884-1952-4323-9dcc-355fb200b2a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:30.424617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424586 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-dshm\") pod \"3a552884-1952-4323-9dcc-355fb200b2a0\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " Oct 31 01:43:30.424750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424615 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-tmp-dir" pod "3a552884-1952-4323-9dcc-355fb200b2a0" (UID: "3a552884-1952-4323-9dcc-355fb200b2a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:30.424750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424633 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-home\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-home\") pod \"3a552884-1952-4323-9dcc-355fb200b2a0\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " Oct 31 01:43:30.424750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424672 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a552884-1952-4323-9dcc-355fb200b2a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a552884-1952-4323-9dcc-355fb200b2a0-tls-certs\") pod \"3a552884-1952-4323-9dcc-355fb200b2a0\" (UID: \"3a552884-1952-4323-9dcc-355fb200b2a0\") " Oct 31 01:43:30.424931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.424909 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-home" pod "3a552884-1952-4323-9dcc-355fb200b2a0" (UID: "3a552884-1952-4323-9dcc-355fb200b2a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:30.425108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.425083 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:30.425108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.425107 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:30.425300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.425116 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:30.426978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.426957 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-dshm" pod "3a552884-1952-4323-9dcc-355fb200b2a0" (UID: "3a552884-1952-4323-9dcc-355fb200b2a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:30.427423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.427406 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a552884-1952-4323-9dcc-355fb200b2a0-tls-certs" pod "3a552884-1952-4323-9dcc-355fb200b2a0" (UID: "3a552884-1952-4323-9dcc-355fb200b2a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:43:30.427488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.427438 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a552884-1952-4323-9dcc-355fb200b2a0-kube-api-access-f24b2" pod "3a552884-1952-4323-9dcc-355fb200b2a0" (UID: "3a552884-1952-4323-9dcc-355fb200b2a0"). InnerVolumeSpecName "kube-api-access-f24b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:43:30.526064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.526033 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a552884-1952-4323-9dcc-355fb200b2a0-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:30.526064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.526063 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a552884-1952-4323-9dcc-355fb200b2a0-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:30.526221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.526079 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f24b2\" (UniqueName: \"kubernetes.io/projected/3a552884-1952-4323-9dcc-355fb200b2a0-kube-api-access-f24b2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:30.851752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:30.851576 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:31.212939 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:31.212868 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" Oct 31 01:43:31.212939 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:31.212909 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b" event={"ID":"3a552884-1952-4323-9dcc-355fb200b2a0","Type":"ContainerDied","Data":"7dca026a799c2b4c90448a1fb6dff1656f9bafb3ee6b355f95a31f70b65d918b"} Oct 31 01:43:31.213434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:31.212963 5090 scope.go:122] "RemoveContainer" containerID="2e9e4c1c9452cc36e0fa081f6dee85accb402856a94cf824702d27c4c01cb9d6" Oct 31 01:43:31.234881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:31.234853 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b"] Oct 31 01:43:31.238994 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:31.238970 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-b-36645eac-kserve-7b6dd8dd54-gkh2b"] Oct 31 01:43:31.464897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:31.464810 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:31.746282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:31.746254 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a552884-1952-4323-9dcc-355fb200b2a0" path="/var/lib/kubelet/pods/3a552884-1952-4323-9dcc-355fb200b2a0/volumes" Oct 31 01:43:32.093292 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:32.093184 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:32.238568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:32.238535 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:33.058319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:33.058262 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:33.747269 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:33.747244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:34.244135 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:34.244107 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:34.457294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:34.457250 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:34.733140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:34.733061 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:34.801140 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:43:34.801104 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:43:35.637774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:35.637707 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" podUID="428c4a53-4c61-448c-af74-bb431b2fad7f" containerName="main" containerID="cri-o://c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50" gracePeriod=45 Oct 31 01:43:35.644497 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:35.644469 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:35.762324 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:35.762286 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" podUID="428c4a53-4c61-448c-af74-bb431b2fad7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused" Oct 31 01:43:35.931892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:35.931870 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:43:36.089155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089117 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-dshm\") pod \"428c4a53-4c61-448c-af74-bb431b2fad7f\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " Oct 31 01:43:36.089343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089181 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-tmp-dir\") pod \"428c4a53-4c61-448c-af74-bb431b2fad7f\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " Oct 31 01:43:36.089343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089223 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-model-cache\") pod \"428c4a53-4c61-448c-af74-bb431b2fad7f\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " Oct 31 01:43:36.089343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089311 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428c4a53-4c61-448c-af74-bb431b2fad7f-kube-api-access-525kd\" (UniqueName: \"kubernetes.io/projected/428c4a53-4c61-448c-af74-bb431b2fad7f-kube-api-access-525kd\") pod \"428c4a53-4c61-448c-af74-bb431b2fad7f\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " Oct 31 01:43:36.089517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089351 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/428c4a53-4c61-448c-af74-bb431b2fad7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/428c4a53-4c61-448c-af74-bb431b2fad7f-tls-certs\") pod \"428c4a53-4c61-448c-af74-bb431b2fad7f\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " Oct 31 01:43:36.089517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089438 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-home\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-home\") pod \"428c4a53-4c61-448c-af74-bb431b2fad7f\" (UID: \"428c4a53-4c61-448c-af74-bb431b2fad7f\") " Oct 31 01:43:36.089623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089513 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-tmp-dir" pod "428c4a53-4c61-448c-af74-bb431b2fad7f" (UID: "428c4a53-4c61-448c-af74-bb431b2fad7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:36.089623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089558 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-model-cache" pod "428c4a53-4c61-448c-af74-bb431b2fad7f" (UID: "428c4a53-4c61-448c-af74-bb431b2fad7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:36.089812 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089788 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-home" pod "428c4a53-4c61-448c-af74-bb431b2fad7f" (UID: "428c4a53-4c61-448c-af74-bb431b2fad7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:36.089937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089922 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:36.089937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089943 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:36.089937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.089959 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:36.091648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.091623 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-dshm" pod "428c4a53-4c61-448c-af74-bb431b2fad7f" (UID: "428c4a53-4c61-448c-af74-bb431b2fad7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:43:36.091790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.091747 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/428c4a53-4c61-448c-af74-bb431b2fad7f-tls-certs" pod "428c4a53-4c61-448c-af74-bb431b2fad7f" (UID: "428c4a53-4c61-448c-af74-bb431b2fad7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:43:36.091881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.091863 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428c4a53-4c61-448c-af74-bb431b2fad7f-kube-api-access-525kd" pod "428c4a53-4c61-448c-af74-bb431b2fad7f" (UID: "428c4a53-4c61-448c-af74-bb431b2fad7f"). InnerVolumeSpecName "kube-api-access-525kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:43:36.191197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.191106 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-525kd\" (UniqueName: \"kubernetes.io/projected/428c4a53-4c61-448c-af74-bb431b2fad7f-kube-api-access-525kd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:36.191197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.191146 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/428c4a53-4c61-448c-af74-bb431b2fad7f-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:36.191197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.191162 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/428c4a53-4c61-448c-af74-bb431b2fad7f-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:43:36.248446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.248414 5090 generic.go:356] "Generic (PLEG): container finished" podID="428c4a53-4c61-448c-af74-bb431b2fad7f" containerID="c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50" exitCode=0 Oct 31 01:43:36.248618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.248469 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" Oct 31 01:43:36.248618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.248498 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" event={"ID":"428c4a53-4c61-448c-af74-bb431b2fad7f","Type":"ContainerDied","Data":"c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50"} Oct 31 01:43:36.248618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.248540 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb" event={"ID":"428c4a53-4c61-448c-af74-bb431b2fad7f","Type":"ContainerDied","Data":"d1931b9ecd96f0cd376c421d1f12ec317dc22acd0a29a5d997857e3943c6df84"} Oct 31 01:43:36.248618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.248559 5090 scope.go:122] "RemoveContainer" containerID="c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50" Oct 31 01:43:36.259091 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.259077 5090 scope.go:122] "RemoveContainer" containerID="c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50" Oct 31 01:43:36.259331 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:43:36.259313 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50\": container with ID starting with c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50 not found: ID does not exist" containerID="c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50" Oct 31 01:43:36.259397 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.259344 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50"} err="failed to get container status \"c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50\": rpc error: code = NotFound desc = could not find container \"c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50\": container with ID starting with c7dde5c47268b92aa0d33bb00fc3e8bff73a507ef694e2bd3e6095083c11cf50 not found: ID does not exist" Oct 31 01:43:36.272370 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.272351 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb"] Oct 31 01:43:36.280108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.280087 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3cf6f4/e2e-conflict-a-36645eac-kserve-86d7cfcbd8-dvrjb"] Oct 31 01:43:36.370625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:36.370588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:37.165342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:37.165181 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:37.748213 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:37.748180 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428c4a53-4c61-448c-af74-bb431b2fad7f" path="/var/lib/kubelet/pods/428c4a53-4c61-448c-af74-bb431b2fad7f/volumes" Oct 31 01:43:37.849525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:37.849398 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:38.184225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.184135 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l"] Oct 31 01:43:38.184638 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.184626 5090 cpu_manager.go:414] "containerMap: removing container" podUID="3a552884-1952-4323-9dcc-355fb200b2a0" containerName="main" Oct 31 01:43:38.184680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.184639 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a552884-1952-4323-9dcc-355fb200b2a0" containerName="main" Oct 31 01:43:38.184680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.184677 5090 cpu_manager.go:414] "containerMap: removing container" podUID="428c4a53-4c61-448c-af74-bb431b2fad7f" containerName="main" Oct 31 01:43:38.184768 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.184682 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428c4a53-4c61-448c-af74-bb431b2fad7f" containerName="main" Oct 31 01:43:38.184808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.184796 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="428c4a53-4c61-448c-af74-bb431b2fad7f" containerName="main" Oct 31 01:43:38.184841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.184809 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a552884-1952-4323-9dcc-355fb200b2a0" containerName="main" Oct 31 01:43:38.189505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.189485 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.205793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.205770 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l"] Oct 31 01:43:38.218817 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.218788 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm"] Oct 31 01:43:38.234752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.231919 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.265309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.265275 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm"] Oct 31 01:43:38.320220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.320184 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:38.325549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325524 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.325745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325556 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.325745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325652 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e2d7e2-60f2-4704-9785-a65450587573-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.325745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325694 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.325893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325779 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.325893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325831 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.325986 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325887 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.325986 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325943 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.325986 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.325976 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7bc789-200b-468c-8525-5b67443e9ed9-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.326147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326098 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.326209 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326148 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e2d7e2-60f2-4704-9785-a65450587573-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.326209 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326178 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.326311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326206 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.326311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326264 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.326410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326319 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.326410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326393 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9g9\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-kube-api-access-wd9g9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.326514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326419 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.326514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.326444 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bql5d\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-kube-api-access-bql5d\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.428054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428000 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.428236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428106 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9g9\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-kube-api-access-wd9g9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.428236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428136 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.428236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428165 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bql5d\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-kube-api-access-bql5d\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.428236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428207 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.428485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428238 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.428485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428264 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e2d7e2-60f2-4704-9785-a65450587573-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.428485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428296 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.428485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428327 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.428485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428362 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.428485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428403 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.428485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428464 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428511 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7bc789-200b-468c-8525-5b67443e9ed9-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428572 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428579 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428614 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e2d7e2-60f2-4704-9785-a65450587573-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428643 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428669 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428748 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.428890 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.429635 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.429138 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.429635 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.429465 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-data\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.429764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.429702 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-credential-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.430505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.429954 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.430505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.430142 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-certs\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.430505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.430205 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-socket\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.430505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.430386 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e2d7e2-60f2-4704-9785-a65450587573-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.431688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.431642 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7bc789-200b-468c-8525-5b67443e9ed9-istiod-ca-cert\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.432273 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.432243 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.433076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.433046 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-envoy\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.433172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.433098 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e2d7e2-60f2-4704-9785-a65450587573-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.434582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.434533 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-podinfo\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.440022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.439972 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9g9\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-kube-api-access-wd9g9\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.440434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.440410 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.440609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.440594 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bql5d\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-kube-api-access-bql5d\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.440681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.440604 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-token\") pod \"e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.511030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.510995 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:38.561460 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.561425 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:38.664149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.664096 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:38.803254 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.803229 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l"] Oct 31 01:43:38.804896 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:43:38.804865 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7bc789_200b_468c_8525_5b67443e9ed9.slice/crio-22e22230579f6cff6620e096f34c958b511d6cead05fc67f1c950ad1044d6783 WatchSource:0}: Error finding container 22e22230579f6cff6620e096f34c958b511d6cead05fc67f1c950ad1044d6783: Status 404 returned error can't find the container with id 22e22230579f6cff6620e096f34c958b511d6cead05fc67f1c950ad1044d6783 Oct 31 01:43:38.807095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.807063 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:38.811513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.807611 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:38.811513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.807771 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:38.883872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.883844 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm"] Oct 31 01:43:38.885794 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:43:38.885771 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e2d7e2_60f2_4704_9785_a65450587573.slice/crio-3ac2fea97e1e75893065e796414afaa61fcffb92b32ddd066ffeac113f99bdcc WatchSource:0}: Error finding container 3ac2fea97e1e75893065e796414afaa61fcffb92b32ddd066ffeac113f99bdcc: Status 404 returned error can't find the container with id 3ac2fea97e1e75893065e796414afaa61fcffb92b32ddd066ffeac113f99bdcc Oct 31 01:43:38.888074 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.888041 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:38.888178 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.888118 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:38.888241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:38.888164 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:39.272142 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.272107 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" event={"ID":"bb7bc789-200b-468c-8525-5b67443e9ed9","Type":"ContainerStarted","Data":"97d8dfc88c65f951a5797f36b4b416e46828a9c1cd12b1ee85f20d58ec8e1e25"} Oct 31 01:43:39.272582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.272164 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" event={"ID":"bb7bc789-200b-468c-8525-5b67443e9ed9","Type":"ContainerStarted","Data":"22e22230579f6cff6620e096f34c958b511d6cead05fc67f1c950ad1044d6783"} Oct 31 01:43:39.273994 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.273968 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" event={"ID":"47e2d7e2-60f2-4704-9785-a65450587573","Type":"ContainerStarted","Data":"2cfea859e5dd5785d83edb02b83bd0bdc229deadedc45577d1ec288dd7c25fb2"} Oct 31 01:43:39.274113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.274024 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" event={"ID":"47e2d7e2-60f2-4704-9785-a65450587573","Type":"ContainerStarted","Data":"3ac2fea97e1e75893065e796414afaa61fcffb92b32ddd066ffeac113f99bdcc"} Oct 31 01:43:39.295162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.295103 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podStartSLOduration=1.295084321 podStartE2EDuration="1.295084321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:39.293804565 +0000 UTC m=+1419.674049810" watchObservedRunningTime="2026-10-31 01:43:39.295084321 +0000 UTC m=+1419.675329546" Oct 31 01:43:39.316904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.316854 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podStartSLOduration=1.316838561 podStartE2EDuration="1.316838561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:39.314984376 +0000 UTC m=+1419.695229599" watchObservedRunningTime="2026-10-31 01:43:39.316838561 +0000 UTC m=+1419.697083783" Oct 31 01:43:39.498741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.498259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:39.512814 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.512762 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:39.534944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.534850 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 01:43:39.535216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.535189 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 01:43:39.562472 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.562445 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:39.563707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.563676 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 01:43:39.563874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:39.563777 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 01:43:40.431057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:40.431021 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:40.520742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:40.518140 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 01:43:40.520742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:40.518204 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 01:43:40.562311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:40.562272 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 01:43:40.562492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:40.562339 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 01:43:41.272623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:41.272567 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:41.511873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:41.511837 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 01:43:41.512464 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:41.512435 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 01:43:41.562039 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:41.561956 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 01:43:41.562039 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:41.562005 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 01:43:42.452691 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:42.452662 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:42.517157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:42.517117 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 01:43:42.517772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:42.517739 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 01:43:42.562616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:42.562576 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 01:43:42.562810 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:42.562633 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 01:43:43.377743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:43.376290 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:43.512177 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:43.512139 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 01:43:43.512436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:43.512402 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 01:43:43.561828 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:43.561791 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 01:43:43.562287 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:43.561863 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 01:43:44.342553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:44.342471 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:44.514739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:44.514448 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 01:43:44.514739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:44.514515 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 01:43:44.561981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:44.561951 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 01:43:44.562465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:44.562430 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 01:43:45.182612 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:45.182477 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:45.511769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:45.511733 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 31 01:43:45.511951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:45.511833 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 31 01:43:45.561944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:45.561915 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 01:43:45.562088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:45.561972 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 01:43:45.994804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:45.986395 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:47.525742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:47.524842 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:47.525742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:47.524949 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:47.576744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:47.574068 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:47.576744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:47.574152 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:47.904131 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:47.903931 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:48.894741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:48.845957 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:48.988842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:48.988790 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:49.088128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:48.913668 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:49.108010 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:49.107968 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:49.118065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:48.956813 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:49.118421 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:49.118291 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:49.665742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:49.665314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:50.011805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.011773 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:50.144895 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.144847 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:50.145594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.145176 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:50.166170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.166139 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:43:50.166501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.166473 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:43:50.453792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.451114 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:50.457998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.456026 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:50.571760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.566210 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:50.578569 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.578542 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 01:43:50.628305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.628274 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:50.629084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.629058 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:50.630099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:50.630075 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 01:43:51.377482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:51.377440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:52.227442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:52.227403 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:52.941844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:52.941808 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:53.640463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:53.640430 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:54.343062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:54.343019 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:55.026034 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:55.025988 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:55.867383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:55.867346 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:56.583310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.583034 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v"] Oct 31 01:43:56.608011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.607978 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v"] Oct 31 01:43:56.608312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.608279 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp"] Oct 31 01:43:56.610405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.610384 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.645450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.645402 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp"] Oct 31 01:43:56.645857 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.645837 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.654624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.654593 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-kube-api-access-xzvsm\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.654756 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.654668 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.656992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.654755 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.657115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.657092 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.657194 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.657134 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3cbc25-a030-4045-8329-58df5508742e-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.657194 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.657169 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.657307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.657198 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.657307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.657279 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3cbc25-a030-4045-8329-58df5508742e-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.657424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.657359 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.758173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758144 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.758363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758181 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3cbc25-a030-4045-8329-58df5508742e-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.758363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758217 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.758363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758241 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.758363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758271 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995px\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-kube-api-access-995px\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.758363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758333 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3cbc25-a030-4045-8329-58df5508742e-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758365 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758408 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758439 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758465 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758508 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758547 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758572 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e2b35e-3e13-4c16-8703-21ff1860286f-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.758645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758603 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-kube-api-access-xzvsm\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.759069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758649 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.759069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758686 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.759069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758740 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.759069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.758816 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.759284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.759258 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.759548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.759516 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.759931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.759906 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.760153 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.760126 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.760842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.760816 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3cbc25-a030-4045-8329-58df5508742e-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.762398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.762360 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.762556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.762538 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3cbc25-a030-4045-8329-58df5508742e-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.775859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.775831 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.800206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.793999 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-kube-api-access-xzvsm\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.859526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859440 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.859526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859496 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.859526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859524 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859549 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859596 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859620 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e2b35e-3e13-4c16-8703-21ff1860286f-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859673 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859765 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859831 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995px\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-kube-api-access-995px\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.859905 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.860184 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.860734 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.860866 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.860845 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e2b35e-3e13-4c16-8703-21ff1860286f-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.861403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.861378 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.863686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.863654 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.863952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.863925 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.872206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.872164 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-995px\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-kube-api-access-995px\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.876767 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.873149 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:56.948741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.945640 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:56.971960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:56.968094 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:57.166790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:57.157581 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:58.068297 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.068259 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v"] Oct 31 01:43:58.153789 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:43:58.133210 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e2b35e_3e13_4c16_8703_21ff1860286f.slice/crio-a70cc7c293c04b7ac53c6ebf7ffc8b8b999ec0e7574166ac7ac0ef61ff78cd05 WatchSource:0}: Error finding container a70cc7c293c04b7ac53c6ebf7ffc8b8b999ec0e7574166ac7ac0ef61ff78cd05: Status 404 returned error can't find the container with id a70cc7c293c04b7ac53c6ebf7ffc8b8b999ec0e7574166ac7ac0ef61ff78cd05 Oct 31 01:43:58.153789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.135030 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp"] Oct 31 01:43:58.153789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.148491 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:58.153789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.148580 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:58.153789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.148655 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:58.203913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.203850 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:58.204306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.204274 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:58.204524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.204483 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:43:58.637036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.636948 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" event={"ID":"5c3cbc25-a030-4045-8329-58df5508742e","Type":"ContainerStarted","Data":"ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa"} Oct 31 01:43:58.642459 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.642434 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" event={"ID":"5c3cbc25-a030-4045-8329-58df5508742e","Type":"ContainerStarted","Data":"9c94be87c95a36bf66a2f87c4e36ffbbbf747564e1fa67de4e9d195ff4eb6d67"} Oct 31 01:43:58.642663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.642645 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" event={"ID":"63e2b35e-3e13-4c16-8703-21ff1860286f","Type":"ContainerStarted","Data":"2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85"} Oct 31 01:43:58.642802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.642786 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" event={"ID":"63e2b35e-3e13-4c16-8703-21ff1860286f","Type":"ContainerStarted","Data":"a70cc7c293c04b7ac53c6ebf7ffc8b8b999ec0e7574166ac7ac0ef61ff78cd05"} Oct 31 01:43:58.676561 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.672736 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podStartSLOduration=2.672695591 podStartE2EDuration="2.672695591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:58.660164151 +0000 UTC m=+1439.040409375" watchObservedRunningTime="2026-10-31 01:43:58.672695591 +0000 UTC m=+1439.052940814" Oct 31 01:43:58.688123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.688073 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podStartSLOduration=2.6880563410000002 podStartE2EDuration="2.688056341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:43:58.688018864 +0000 UTC m=+1439.068264112" watchObservedRunningTime="2026-10-31 01:43:58.688056341 +0000 UTC m=+1439.068301565" Oct 31 01:43:58.722107 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.722049 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:58.947124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.947042 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:43:58.969461 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:58.969434 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:43:59.632736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:59.632610 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:43:59.948834 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:59.948803 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:43:59.948966 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:59.948878 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 01:43:59.970572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:59.970542 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 01:43:59.970652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:43:59.970583 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 01:44:00.535173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:00.534834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:00.946637 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:00.946556 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:44:00.946637 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:00.946604 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 01:44:00.969671 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:00.969648 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 01:44:00.969796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:00.969702 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 01:44:01.417742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:01.417422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:01.946778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:01.946737 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:44:01.947165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:01.946839 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 01:44:01.969329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:01.969296 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 01:44:01.969489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:01.969394 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 01:44:02.261675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:02.261644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:02.947160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:02.947127 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:44:02.947606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:02.947201 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 01:44:02.968844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:02.968812 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 01:44:02.968978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:02.968876 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 01:44:03.090755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:03.090131 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:03.446203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:03.446163 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:03.947097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:03.947060 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:44:03.947306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:03.947115 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 01:44:03.969038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:03.969000 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 01:44:03.969198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:03.969042 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 01:44:04.130738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.128634 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:04.790878 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:44:04.790846 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:44:04.829990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.829959 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd"] Oct 31 01:44:04.836273 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.836248 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.860338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.860301 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd"] Oct 31 01:44:04.945282 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945234 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfr2\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-kube-api-access-qrfr2\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945312 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945421 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945449 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945598 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945681 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945706 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945795 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.945962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.945843 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:04.946729 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.946689 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 31 01:44:04.946818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.946762 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 31 01:44:04.969335 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.969302 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 01:44:04.969865 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:04.969361 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 01:44:05.046623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046530 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046586 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046621 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046663 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046685 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046731 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046774 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046828 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.046944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.046916 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfr2\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-kube-api-access-qrfr2\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.047252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.047119 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.047252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.047183 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.047588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.047419 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.047588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.047549 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.048028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.047998 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.049554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.049533 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.049758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.049736 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.061243 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.061174 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.065945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.063943 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfr2\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-kube-api-access-qrfr2\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.065945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.064705 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:05.157991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.157952 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:05.464344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.464311 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd"] Oct 31 01:44:05.466438 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:44:05.466413 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429afc9a_4c05_4dac_ba33_e1e25c13ad9c.slice/crio-f60d9f12e3acf18a7ab9327b0581acfadfeece6bfaba2e49a2845938ccb5fff9 WatchSource:0}: Error finding container f60d9f12e3acf18a7ab9327b0581acfadfeece6bfaba2e49a2845938ccb5fff9: Status 404 returned error can't find the container with id f60d9f12e3acf18a7ab9327b0581acfadfeece6bfaba2e49a2845938ccb5fff9 Oct 31 01:44:05.469455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.469415 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:05.469534 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.469508 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:05.469572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.469555 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:05.711837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.708417 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" event={"ID":"429afc9a-4c05-4dac-ba33-e1e25c13ad9c","Type":"ContainerStarted","Data":"f62e1d70184d1c7bd3c84a61991b8f7d04ef3a52753d800ccabe0f565d152da8"} Oct 31 01:44:05.711837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.708502 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" event={"ID":"429afc9a-4c05-4dac-ba33-e1e25c13ad9c","Type":"ContainerStarted","Data":"f60d9f12e3acf18a7ab9327b0581acfadfeece6bfaba2e49a2845938ccb5fff9"} Oct 31 01:44:05.736791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.736495 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podStartSLOduration=1.736476683 podStartE2EDuration="1.736476683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:05.729566114 +0000 UTC m=+1446.109811366" watchObservedRunningTime="2026-10-31 01:44:05.736476683 +0000 UTC m=+1446.116721909" Oct 31 01:44:05.911737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:05.911676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:06.158991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.158955 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:06.160694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.160664 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 01:44:06.160940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.160761 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 01:44:06.172588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.172558 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:44:06.172979 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.172950 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:44:06.173778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.173757 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:44:06.194894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.194866 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:44:06.195321 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.195295 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:44:06.196674 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.196652 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 01:44:06.670268 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:06.670220 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:07.159258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:07.159223 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 01:44:07.159659 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:07.159286 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 01:44:07.562740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:07.554274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:08.159739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:08.159216 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 01:44:08.159739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:08.159318 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 01:44:09.164763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.160135 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 01:44:09.164763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.160196 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 01:44:09.283742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.277270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:09.583682 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.583645 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7"] Oct 31 01:44:09.606741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.597831 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.612889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.612852 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7"] Oct 31 01:44:09.697461 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.697417 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6075252e-be6b-444f-99d6-68292c59f906-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.698874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.698823 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-998tr\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-kube-api-access-998tr\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.699027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.698897 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.699027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.698957 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.699130 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.699113 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.699179 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.699152 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.699329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.699309 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6075252e-be6b-444f-99d6-68292c59f906-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.699384 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.699366 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.699475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.699456 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801075 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801155 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801231 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801265 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801364 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6075252e-be6b-444f-99d6-68292c59f906-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801413 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801458 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801493 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6075252e-be6b-444f-99d6-68292c59f906-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.801556 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-998tr\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-kube-api-access-998tr\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.802403 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.802879 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6075252e-be6b-444f-99d6-68292c59f906-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.803252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.803041 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.804280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.804255 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.804807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.804786 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.806154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.806118 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6075252e-be6b-444f-99d6-68292c59f906-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.807087 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.807065 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.814772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.814748 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.821041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.821007 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-998tr\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-kube-api-access-998tr\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:09.928222 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:09.928141 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:10.159032 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.158994 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 01:44:10.159199 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.159053 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 01:44:10.210802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.210610 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:10.249994 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.249964 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7"] Oct 31 01:44:10.250531 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:44:10.250504 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6075252e_be6b_444f_99d6_68292c59f906.slice/crio-320152e5f6307d11d37cb53021d15ca3843aafd68a952d3b0a906f7257e8160f WatchSource:0}: Error finding container 320152e5f6307d11d37cb53021d15ca3843aafd68a952d3b0a906f7257e8160f: Status 404 returned error can't find the container with id 320152e5f6307d11d37cb53021d15ca3843aafd68a952d3b0a906f7257e8160f Oct 31 01:44:10.253409 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.253369 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:10.253520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.253455 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:10.253520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.253494 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:10.743946 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.743873 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:10.763831 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.763765 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" event={"ID":"6075252e-be6b-444f-99d6-68292c59f906","Type":"ContainerStarted","Data":"eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73"} Oct 31 01:44:10.764004 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.763910 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" event={"ID":"6075252e-be6b-444f-99d6-68292c59f906","Type":"ContainerStarted","Data":"320152e5f6307d11d37cb53021d15ca3843aafd68a952d3b0a906f7257e8160f"} Oct 31 01:44:10.928943 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:10.928904 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:11.159215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:11.159134 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 01:44:11.159215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:11.159189 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 01:44:11.213751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:11.210738 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:11.930928 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:11.930883 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:44:11.931093 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:11.930972 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 01:44:12.151235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:12.151203 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:12.159050 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:12.159019 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 01:44:12.159231 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:12.159074 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 01:44:12.928895 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:12.928856 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:44:12.929442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:12.928919 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 01:44:12.944069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:12.944030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:13.212897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:13.212802 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:13.214614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:13.214590 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:13.216193 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:13.215784 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 01:44:13.238881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:13.238830 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podStartSLOduration=4.238810536 podStartE2EDuration="4.238810536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:10.803554643 +0000 UTC m=+1451.183799884" watchObservedRunningTime="2026-10-31 01:44:13.238810536 +0000 UTC m=+1453.619055758" Oct 31 01:44:13.672186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:13.672153 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:13.929398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:13.929318 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:44:13.929398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:13.929375 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 01:44:14.464984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:14.464955 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:14.928885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:14.928847 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:44:14.929096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:14.928920 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 01:44:15.263226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:15.263195 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:15.928597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:15.928559 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:44:15.928791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:15.928619 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 01:44:15.962553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:15.962526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:16.674334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:16.674302 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:16.929127 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:16.929041 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 31 01:44:16.929127 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:16.929103 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 31 01:44:17.384957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:17.384915 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:17.933213 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:17.933183 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:17.933703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:17.933679 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:17.934468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:17.934452 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 01:44:18.079226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:18.079193 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:18.539465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:18.539437 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:19.126930 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:19.126894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:19.881570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:19.875309 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9"] Oct 31 01:44:19.888204 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:19.888178 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:19.923115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:19.913251 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9"] Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.013674 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.013750 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5ace9-0f21-4450-b319-69b5064c4336-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.013780 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5ace9-0f21-4450-b319-69b5064c4336-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.013912 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.013945 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.014090 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th9xs\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-kube-api-access-th9xs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.014127 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.014210 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.014297 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.121516 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.121468 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.121810 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.121786 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.122024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.122007 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th9xs\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-kube-api-access-th9xs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.122283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.122263 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.122444 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.122426 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.122607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.122590 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.122747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.122709 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.142795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.122851 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5ace9-0f21-4450-b319-69b5064c4336-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.143406 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.143368 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5ace9-0f21-4450-b319-69b5064c4336-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.144556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.144531 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5ace9-0f21-4450-b319-69b5064c4336-istiod-ca-cert\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.145010 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.144990 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.145351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.145328 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-data\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.150784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.145360 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5ace9-0f21-4450-b319-69b5064c4336-istio-podinfo\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.150910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.145765 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-certs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.151038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.146608 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-credential-socket\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.199580 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.199154 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th9xs\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-kube-api-access-th9xs\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.203759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.203235 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-istio-token\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.203759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.203681 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:20.211745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.210759 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-envoy\") pod \"e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:20.416351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:20.416004 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:21.736843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:21.735114 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:21.736843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:21.735173 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:21.983742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:21.968969 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9"] Oct 31 01:44:22.121779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:22.111674 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:22.121779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:22.111800 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:22.121779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:22.111850 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:22.494902 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:22.493755 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:23.077606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:23.077556 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" event={"ID":"34c5ace9-0f21-4450-b319-69b5064c4336","Type":"ContainerStarted","Data":"b3752f47ecb5cfc5f65a7b6166b624109e270882f555d31492d40c6517e51b2a"} Oct 31 01:44:23.078176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:23.077635 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" event={"ID":"34c5ace9-0f21-4450-b319-69b5064c4336","Type":"ContainerStarted","Data":"27654ac3606b17d7a22867cb2a534a426e08c8fc29d62a67466d70e5ede381c0"} Oct 31 01:44:23.111583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:23.106896 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podStartSLOduration=4.106879788 podStartE2EDuration="4.106879788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:23.100687808 +0000 UTC m=+1463.480933058" watchObservedRunningTime="2026-10-31 01:44:23.106879788 +0000 UTC m=+1463.487125010" Oct 31 01:44:23.366673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:23.366587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:23.420901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:23.420865 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:23.422466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:23.422445 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:44:23.422525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:23.422494 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 01:44:24.366554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.366511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:24.421765 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.421694 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:44:24.421936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.421777 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 01:44:24.646760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.646664 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg"] Oct 31 01:44:24.651023 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.651001 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.664463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.664440 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg"] Oct 31 01:44:24.747655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.747609 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.747882 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.747780 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.747960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.747892 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.747960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.747943 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.748082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.747991 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.748082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.748052 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nt7b\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-kube-api-access-4nt7b\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.748213 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.748194 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.748258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.748244 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.748329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.748318 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849092 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849059 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849112 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849153 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849180 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849240 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849287 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849332 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849361 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849412 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nt7b\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-kube-api-access-4nt7b\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849691 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849572 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849691 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849609 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-credential-socket\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849881 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-data\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.849962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.849939 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-certs\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.850080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.850061 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istiod-ca-cert\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.851908 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.851885 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-envoy\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.852213 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.852197 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-podinfo\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.860899 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.860872 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-token\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.860991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.860970 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nt7b\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-kube-api-access-4nt7b\") pod \"e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:24.965092 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:24.965009 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:25.308582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:25.308555 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg"] Oct 31 01:44:25.311065 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:44:25.310858 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01ed085_5ab8_4f9e_ab11_5d40f3c14652.slice/crio-2270584c8c7883cafc0f3c102f9468e7ed271240df3d739e715e77e0c0f89a22 WatchSource:0}: Error finding container 2270584c8c7883cafc0f3c102f9468e7ed271240df3d739e715e77e0c0f89a22: Status 404 returned error can't find the container with id 2270584c8c7883cafc0f3c102f9468e7ed271240df3d739e715e77e0c0f89a22 Oct 31 01:44:25.313996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:25.313968 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:25.314064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:25.314038 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:25.314111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:25.314077 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:25.316884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:25.316319 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:25.420196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:25.420167 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:44:25.420486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:25.420215 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 01:44:26.162066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.162017 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" event={"ID":"e01ed085-5ab8-4f9e-ab11-5d40f3c14652","Type":"ContainerStarted","Data":"7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1"} Oct 31 01:44:26.162335 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.162316 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" event={"ID":"e01ed085-5ab8-4f9e-ab11-5d40f3c14652","Type":"ContainerStarted","Data":"2270584c8c7883cafc0f3c102f9468e7ed271240df3d739e715e77e0c0f89a22"} Oct 31 01:44:26.194742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.194060 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podStartSLOduration=2.194020869 podStartE2EDuration="2.194020869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:26.187369388 +0000 UTC m=+1466.567614611" watchObservedRunningTime="2026-10-31 01:44:26.194020869 +0000 UTC m=+1466.574266093" Oct 31 01:44:26.222743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.222425 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:26.423207 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.423114 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:44:26.423802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.423769 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 01:44:26.965516 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.965484 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:26.967434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.967405 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:44:26.967560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:26.967460 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 01:44:27.059629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:27.059588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:27.420306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:27.420272 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:44:27.420579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:27.420550 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 01:44:27.960995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:27.960963 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:27.967226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:27.965704 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:44:27.967488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:27.967457 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 01:44:28.424124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:28.424086 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:44:28.424432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:28.424396 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 01:44:28.833735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:28.830495 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:28.966049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:28.966009 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:44:28.966427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:28.966072 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 01:44:29.420516 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:29.420480 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 31 01:44:29.420734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:29.420549 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 31 01:44:29.896474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:29.896345 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:29.965688 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:29.965656 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:44:29.965871 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:29.965709 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 01:44:30.425485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:30.425456 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:30.425986 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:30.425894 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:30.426845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:30.426827 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 01:44:30.853199 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:30.853159 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:30.966326 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:30.966293 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:44:30.966492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:30.966353 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 01:44:31.754740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:31.747366 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:31.966359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:31.966326 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 01:44:31.966569 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:31.966387 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 01:44:32.567601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:32.567566 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:33.973857 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:33.973815 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:33.974478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:33.974445 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:35.033743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:35.029828 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:35.033743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:35.029900 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:35.582748 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:44:35.579934 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:44:35.812793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:35.807177 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:35.842945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:35.842854 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:36.123797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:36.119809 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:36.123797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:36.119876 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:36.628746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:36.626745 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:36.628746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:36.626826 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:36.763742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:36.762856 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:36.763742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:36.762984 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:37.334738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:37.307140 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:37.425833 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:37.358969 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:37.426138 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:37.426101 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:37.426596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:37.371178 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:37.428490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:37.428462 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:37.449021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:37.426951 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:38.469818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:38.467114 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:38.469818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:38.467193 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:38.476409 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:38.475508 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:38.476847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:38.476595 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:38.479114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:38.479096 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:44:38.566111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:38.566078 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:39.443928 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:39.443894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:39.840924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:39.840887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:40.345589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:40.345550 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:41.322621 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:41.322587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:42.047372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:42.047331 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:42.672866 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:42.672776 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:43.350388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:43.350355 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:44.062030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:44.060659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:44.761351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:44.754088 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:45.451325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:45.450530 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:46.230560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:46.230515 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:47.055742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:47.050609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:47.755277 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:47.755230 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:48.359445 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:48.359410 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:48.977914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:48.977879 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:49.744646 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:49.744587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:50.024576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:50.024492 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:50.168985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:50.168949 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:50.453888 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:50.453789 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:51.477171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:51.477138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:54.572745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:54.557057 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:44:54.572745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:54.557196 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:44:55.090790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.057475 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85"] Oct 31 01:44:55.161294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.161265 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.354112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.354010 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85"] Oct 31 01:44:55.387995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.387943 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.388272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.388239 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.388462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.388432 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.388605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.388587 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.388786 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.388755 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d5940c-4db0-471a-a110-c91ff831ae1b-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.402388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.402354 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkvc\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-kube-api-access-cmkvc\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.402675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.402637 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.402985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.402964 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.403179 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.403150 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505179 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505145 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505191 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505339 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505361 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505391 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505422 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505460 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d5940c-4db0-471a-a110-c91ff831ae1b-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505491 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkvc\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-kube-api-access-cmkvc\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505531 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.505908 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505753 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.506015 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.505901 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.506158 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.506137 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.506423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.506378 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.507517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.507478 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d5940c-4db0-471a-a110-c91ff831ae1b-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.509416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.509392 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.513263 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.513242 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.518205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.518166 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkvc\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-kube-api-access-cmkvc\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.519070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.518955 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.606869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.605864 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:55.688553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.688496 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:55.982022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.982001 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85"] Oct 31 01:44:55.984556 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:44:55.984523 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d5940c_4db0_471a_a110_c91ff831ae1b.slice/crio-8ae7ac5c73832c2595e01897a9483f66fc89b0be2376a614b8a62b7f9782f8ce WatchSource:0}: Error finding container 8ae7ac5c73832c2595e01897a9483f66fc89b0be2376a614b8a62b7f9782f8ce: Status 404 returned error can't find the container with id 8ae7ac5c73832c2595e01897a9483f66fc89b0be2376a614b8a62b7f9782f8ce Oct 31 01:44:55.987236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.987185 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:55.987363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.987285 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:55.987363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:55.987329 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:44:56.362621 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:56.362585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:56.738620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:56.738535 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" event={"ID":"26d5940c-4db0-471a-a110-c91ff831ae1b","Type":"ContainerStarted","Data":"3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005"} Oct 31 01:44:56.739168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:56.738631 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" event={"ID":"26d5940c-4db0-471a-a110-c91ff831ae1b","Type":"ContainerStarted","Data":"8ae7ac5c73832c2595e01897a9483f66fc89b0be2376a614b8a62b7f9782f8ce"} Oct 31 01:44:56.770759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:56.770678 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podStartSLOduration=2.770657834 podStartE2EDuration="2.770657834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:44:56.769347375 +0000 UTC m=+1497.149592617" watchObservedRunningTime="2026-10-31 01:44:56.770657834 +0000 UTC m=+1497.150903057" Oct 31 01:44:57.264982 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:57.264950 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:57.438703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:57.438647 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:57.600272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:57.600177 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:57.696903 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:57.696785 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:44:57.699527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:57.699164 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:44:57.699527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:57.699225 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 01:44:57.745546 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:57.745512 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:58.135623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:58.135592 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:58.700886 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:58.700855 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:44:58.701168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:58.701140 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 01:44:58.703110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:58.703073 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:58.852925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:58.852892 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:59.002137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.002105 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:59.153705 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.153672 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:44:59.697770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.691264 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:44:59.697770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.691312 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 01:44:59.930167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.930141 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:44:59.930913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.930895 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:44:59.937176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.937157 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:44:59.939292 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:44:59.939272 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:45:00.054743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:00.047980 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:00.698218 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:00.698176 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:45:00.698433 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:00.698244 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 01:45:00.794680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:00.783348 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:01.568102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:01.568060 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:01.690222 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:01.690191 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:45:01.690518 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:01.690489 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 01:45:02.450565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:02.450534 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:02.689529 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:02.689491 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 31 01:45:02.689960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:02.689546 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 31 01:45:03.274775 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:03.272279 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:03.694066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:03.693988 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:45:03.694515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:03.694283 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:45:03.695027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:03.694997 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 01:45:04.049558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:04.049399 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:04.791229 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:45:04.791198 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:45:04.851270 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:04.851224 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:05.743433 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:05.743379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:07.295361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:07.295330 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:07.977384 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:07.977208 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:08.517978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:08.517933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:09.186482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:09.186445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:09.528148 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:09.526123 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:45:09.528148 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:09.526209 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:45:09.752741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:09.752234 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:10.838657 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:10.838622 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:11.251357 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:11.251324 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:12.074060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:12.074014 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:12.918817 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:12.918782 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:13.638764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:13.638707 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:13.786157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:13.786108 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:14.580740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:14.571132 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:14.882639 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:14.882565 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:15.109308 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:15.109278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:15.653745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:15.652468 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:16.351652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:16.351617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:17.223903 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.223867 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:17.858286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.858243 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j"] Oct 31 01:45:17.871676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.871644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:17.872173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.872065 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j"] Oct 31 01:45:17.872426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.872410 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:17.875764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.875741 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cb5756cb-serving-cert\"" Oct 31 01:45:17.982740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.980013 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-maas-api-tls\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:17.982740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.980115 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-metrics-certs\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:17.982740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.980179 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qjbb\" (UniqueName: \"kubernetes.io/projected/ede888a8-8881-464a-8706-972e92fa52ba-kube-api-access-2qjbb\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:17.982740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:17.980440 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede888a8-8881-464a-8706-972e92fa52ba-openshift-service-ca\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.023111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.023066 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:18.081880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.081841 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede888a8-8881-464a-8706-972e92fa52ba-openshift-service-ca\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.082099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.081959 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-maas-api-tls\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.082099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.082007 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-metrics-certs\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.082099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.082069 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qjbb\" (UniqueName: \"kubernetes.io/projected/ede888a8-8881-464a-8706-972e92fa52ba-kube-api-access-2qjbb\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.082824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.082796 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede888a8-8881-464a-8706-972e92fa52ba-openshift-service-ca\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.085064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.085040 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-metrics-certs\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.085291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.085271 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-maas-api-tls\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.100462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.100439 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qjbb\" (UniqueName: \"kubernetes.io/projected/ede888a8-8881-464a-8706-972e92fa52ba-kube-api-access-2qjbb\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.175424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.175351 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:18.187511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.187486 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:18.469924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.469888 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j"] Oct 31 01:45:18.471753 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:45:18.471694 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede888a8_8881_464a_8706_972e92fa52ba.slice/crio-7b561864283afd6e2f2248aa5719a0406348ed05355354ac8ae66b9e07f91034 WatchSource:0}: Error finding container 7b561864283afd6e2f2248aa5719a0406348ed05355354ac8ae66b9e07f91034: Status 404 returned error can't find the container with id 7b561864283afd6e2f2248aa5719a0406348ed05355354ac8ae66b9e07f91034 Oct 31 01:45:18.940259 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:18.937168 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:19.302242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:19.302203 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" event={"ID":"ede888a8-8881-464a-8706-972e92fa52ba","Type":"ContainerStarted","Data":"3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876"} Oct 31 01:45:19.302429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:19.302258 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" event={"ID":"ede888a8-8881-464a-8706-972e92fa52ba","Type":"ContainerStarted","Data":"7b561864283afd6e2f2248aa5719a0406348ed05355354ac8ae66b9e07f91034"} Oct 31 01:45:19.302501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:19.302423 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:19.322571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:19.322526 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" podStartSLOduration=1.646072757 podStartE2EDuration="2.32251137s" totalImagesPullingTime="676.438613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:19.32050903 +0000 UTC m=+1519.700754277" watchObservedRunningTime="2026-10-31 01:45:19.32251137 +0000 UTC m=+1519.702756594" Oct 31 01:45:19.637594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:19.637552 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:20.278299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:20.278244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:20.990150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:20.990104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:21.731197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:21.731167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:22.572815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:22.561260 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:23.260701 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:23.260664 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:23.877366 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:23.873185 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:24.650520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:24.648009 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:25.317066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:25.317037 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:45:25.331808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:25.331771 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:25.960191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:25.960138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:26.642340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:26.642286 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:27.402075 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:27.402038 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:28.048361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:28.047743 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:28.852763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:28.848405 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:29.486758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:29.484406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:30.159245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:30.159182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:30.905133 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:30.905019 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:31.813931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:31.813894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:33.538742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:33.529274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:33.732741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:33.722875 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:33.868793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:33.868679 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:34.062742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:34.058323 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:35.396737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:35.388976 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:35.572784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:35.567913 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:35.825697 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:45:35.825661 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:45:35.994565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:35.994533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:36.688863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:36.688825 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:45:36.689446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:36.689413 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:45:36.696211 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:36.696160 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:45:36.696453 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:36.696417 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:45:37.618881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:37.618812 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:37.924968 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:37.924836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:39.266031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:39.265969 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:40.023693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:40.023658 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:40.236876 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:40.236844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:40.855923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:40.855891 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:41.674662 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:41.674630 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:42.745671 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:42.745632 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:43.456048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:43.456002 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:43.597306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:43.597265 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:43.787602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:43.787560 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:43.943156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:43.943127 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:44.494183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:44.494151 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:44.782346 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:44.782319 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:45.029320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:45.029235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:45.563889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:45.559138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:45.827572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:45.827500 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:45.998798 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:45.978604 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:46.441005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:46.440971 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:47.326599 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:47.326542 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:48.097336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.097299 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:48.746223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.746193 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw"] Oct 31 01:45:48.774115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.774086 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.776951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.776924 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw"] Oct 31 01:45:48.778316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.778291 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1b634e-serving-cert\"" Oct 31 01:45:48.832588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.832560 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:48.839041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.839015 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdj8d\" (UniqueName: \"kubernetes.io/projected/505db6a7-31d6-49a7-892b-07cfc82f5caa-kube-api-access-tdj8d\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.839189 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.839166 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-metrics-certs\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.839285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.839268 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-maas-api-tls\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.839342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.839327 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505db6a7-31d6-49a7-892b-07cfc82f5caa-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.941639 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.941602 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdj8d\" (UniqueName: \"kubernetes.io/projected/505db6a7-31d6-49a7-892b-07cfc82f5caa-kube-api-access-tdj8d\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.942080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.942047 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-metrics-certs\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.942196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.942147 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-maas-api-tls\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.942262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.942211 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505db6a7-31d6-49a7-892b-07cfc82f5caa-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.943216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.943141 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505db6a7-31d6-49a7-892b-07cfc82f5caa-openshift-service-ca\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.945218 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.945191 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-maas-api-tls\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.945330 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.945303 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-metrics-certs\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.954514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.954485 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdj8d\" (UniqueName: \"kubernetes.io/projected/505db6a7-31d6-49a7-892b-07cfc82f5caa-kube-api-access-tdj8d\") pod \"maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:48.971547 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:48.971504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:49.092500 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.092417 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:49.121749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.121683 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:49.380112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.380082 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw"] Oct 31 01:45:49.382309 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:45:49.382279 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505db6a7_31d6_49a7_892b_07cfc82f5caa.slice/crio-1b878bcef8da2ea9d0c8ac6d36208e8b7021b56d0ddcb2c3e0f95cacef6d6da4 WatchSource:0}: Error finding container 1b878bcef8da2ea9d0c8ac6d36208e8b7021b56d0ddcb2c3e0f95cacef6d6da4: Status 404 returned error can't find the container with id 1b878bcef8da2ea9d0c8ac6d36208e8b7021b56d0ddcb2c3e0f95cacef6d6da4 Oct 31 01:45:49.736398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.736315 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg"] Oct 31 01:45:49.742668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.742636 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.747487 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.747171 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9cc66e-serving-cert\"" Oct 31 01:45:49.771420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.771389 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg"] Oct 31 01:45:49.833917 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.833890 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:49.873537 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.873511 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.873644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.873580 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-metrics-certs\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.874242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.873934 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpwq\" (UniqueName: \"kubernetes.io/projected/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-kube-api-access-kjpwq\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.874242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.874031 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-maas-api-tls\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.979452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.979418 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:49.994791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.994617 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpwq\" (UniqueName: \"kubernetes.io/projected/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-kube-api-access-kjpwq\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.994961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.994814 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-maas-api-tls\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.994961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.994931 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.995078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.995008 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-metrics-certs\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:49.995945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:49.995884 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-openshift-service-ca\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:50.004393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.004364 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-maas-api-tls\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:50.012809 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.005767 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-metrics-certs\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:50.022999 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.022968 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpwq\" (UniqueName: \"kubernetes.io/projected/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-kube-api-access-kjpwq\") pod \"maas-api-e2e-api-a-9cc66e-74d656df45-99czg\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:50.090507 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.090470 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:50.124559 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.124526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:50.218743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.207528 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" event={"ID":"505db6a7-31d6-49a7-892b-07cfc82f5caa","Type":"ContainerStarted","Data":"b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3"} Oct 31 01:45:50.218743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.207613 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" event={"ID":"505db6a7-31d6-49a7-892b-07cfc82f5caa","Type":"ContainerStarted","Data":"1b878bcef8da2ea9d0c8ac6d36208e8b7021b56d0ddcb2c3e0f95cacef6d6da4"} Oct 31 01:45:50.218743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.208579 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:50.269882 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.269647 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" podStartSLOduration=1.807650322 podStartE2EDuration="2.269628473s" totalImagesPullingTime="461.978151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:50.264152873 +0000 UTC m=+1550.644398098" watchObservedRunningTime="2026-10-31 01:45:50.269628473 +0000 UTC m=+1550.649873697" Oct 31 01:45:50.331843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.331804 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:50.475017 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.474983 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg"] Oct 31 01:45:50.475247 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:45:50.475020 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5b46ee_30e5_4a3e_be9b_07322139e1a6.slice/crio-c6f4e840f88c6222dcc0fb94d29a92f2fe8a597f5d8e7e943389f77e3b0aa63b WatchSource:0}: Error finding container c6f4e840f88c6222dcc0fb94d29a92f2fe8a597f5d8e7e943389f77e3b0aa63b: Status 404 returned error can't find the container with id c6f4e840f88c6222dcc0fb94d29a92f2fe8a597f5d8e7e943389f77e3b0aa63b Oct 31 01:45:50.822394 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:50.822325 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:51.204537 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:51.204448 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" event={"ID":"8d5b46ee-30e5-4a3e-be9b-07322139e1a6","Type":"ContainerStarted","Data":"aba8d44f5dd1b2daaa5009d3c172060b65322920091021f406acf2b2b1c13dc5"} Oct 31 01:45:51.204537 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:51.204503 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" event={"ID":"8d5b46ee-30e5-4a3e-be9b-07322139e1a6","Type":"ContainerStarted","Data":"c6f4e840f88c6222dcc0fb94d29a92f2fe8a597f5d8e7e943389f77e3b0aa63b"} Oct 31 01:45:51.204990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:51.204684 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:51.224967 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:51.224912 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" podStartSLOduration=1.751138438 podStartE2EDuration="2.224895227s" totalImagesPullingTime="473.756789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:51.223869735 +0000 UTC m=+1551.604114980" watchObservedRunningTime="2026-10-31 01:45:51.224895227 +0000 UTC m=+1551.605140451" Oct 31 01:45:51.654906 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:51.648858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:52.387095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:52.384794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:53.478193 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:53.478159 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:53.677406 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:53.664133 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:54.454069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:54.454034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:55.160584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.160551 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms"] Oct 31 01:45:55.165891 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.165863 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.197780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.195949 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms"] Oct 31 01:45:55.209911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.209807 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.213829 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.213803 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.214001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.213986 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.214181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.214166 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwv7\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-kube-api-access-brwv7\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.214298 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.214285 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.214418 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.214406 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.214626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.214611 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35804e-37df-4f7a-b0c6-2eb266f31523-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.214750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.214735 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.214862 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.214849 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.316693 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.316811 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwv7\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-kube-api-access-brwv7\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.316856 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.316908 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.317001 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35804e-37df-4f7a-b0c6-2eb266f31523-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.317046 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.317091 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.317222 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.317258 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.320740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.320405 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-credential-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.329737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.325790 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-certs\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.329737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.329183 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-podinfo\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.329737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.329458 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-socket\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.337526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.335041 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-data\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.337526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.335486 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35804e-37df-4f7a-b0c6-2eb266f31523-istiod-ca-cert\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.353256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.353178 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwv7\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-kube-api-access-brwv7\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.355206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.355173 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-token\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.355353 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.355173 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-envoy\") pod \"e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.394563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.394532 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:55.524224 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.524192 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:55.852905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.852858 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms"] Oct 31 01:45:55.854205 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:45:55.854179 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c35804e_37df_4f7a_b0c6_2eb266f31523.slice/crio-cf16a13c91aecb92c681b179d50ba7dc4bbfa31f0546d615a1aeaeee7c982d7e WatchSource:0}: Error finding container cf16a13c91aecb92c681b179d50ba7dc4bbfa31f0546d615a1aeaeee7c982d7e: Status 404 returned error can't find the container with id cf16a13c91aecb92c681b179d50ba7dc4bbfa31f0546d615a1aeaeee7c982d7e Oct 31 01:45:55.856894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.856860 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:45:55.857018 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.856946 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:45:55.857083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:55.857016 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:45:56.092839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:56.092745 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:56.245795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:56.245756 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" event={"ID":"3c35804e-37df-4f7a-b0c6-2eb266f31523","Type":"ContainerStarted","Data":"faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d"} Oct 31 01:45:56.246239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:56.245811 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" event={"ID":"3c35804e-37df-4f7a-b0c6-2eb266f31523","Type":"ContainerStarted","Data":"cf16a13c91aecb92c681b179d50ba7dc4bbfa31f0546d615a1aeaeee7c982d7e"} Oct 31 01:45:56.273243 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:56.273187 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podStartSLOduration=1.273167832 podStartE2EDuration="1.273167832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:45:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:45:56.268629812 +0000 UTC m=+1556.648875035" watchObservedRunningTime="2026-10-31 01:45:56.273167832 +0000 UTC m=+1556.653413055" Oct 31 01:45:56.524438 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:56.524351 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:45:56.526115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:56.526082 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 01:45:56.526254 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:56.526144 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 01:45:57.164767 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:57.164688 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:57.236759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:57.234486 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:45:57.236759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:57.236117 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:45:57.525170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:57.525137 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 01:45:57.525739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:57.525196 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 01:45:58.437750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:58.436259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:58.525016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:58.524979 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 01:45:58.525187 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:58.525033 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 01:45:58.769021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:58.768990 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:59.321957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:59.321922 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:59.497213 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:59.497180 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:45:59.524674 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:59.524637 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 01:45:59.524899 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:45:59.524705 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 01:46:00.015094 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:00.015055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:00.403793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:00.401377 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:00.526154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:00.526119 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 01:46:00.526556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:00.526522 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 01:46:00.545823 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:00.545746 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:01.273742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:01.271413 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:01.525091 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:01.525006 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 01:46:01.525091 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:01.525062 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 01:46:02.075732 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:02.075685 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:02.525220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:02.525183 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 31 01:46:02.525733 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:02.525239 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 31 01:46:02.852806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:02.848000 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:03.509206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:03.509166 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:03.530333 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:03.530301 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:46:03.530854 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:03.530822 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:46:03.531665 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:03.531644 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 01:46:04.053589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:04.053561 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:04.234109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:04.234074 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:04.399391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:04.399307 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:04.790301 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:46:04.790262 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:46:05.301602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:05.301564 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:05.897426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:05.897385 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:06.081740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:06.080139 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:06.297770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:06.297737 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:06.957317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:06.957271 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:07.589901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:07.589825 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:08.235862 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:08.235788 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:08.799376 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:08.799344 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:08.944532 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:08.944501 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:09.452647 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:09.452614 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:09.592979 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:09.592949 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:09.900384 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:09.900329 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:10.208181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.208055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:10.348244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.348015 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt"] Oct 31 01:46:10.355418 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.355392 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.363492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.363468 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b6b833-serving-cert\"" Oct 31 01:46:10.369170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.369146 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt"] Oct 31 01:46:10.498176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.498142 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbnt\" (UniqueName: \"kubernetes.io/projected/dc1ebc11-1e71-41c3-ab12-e33c21661cce-kube-api-access-bjbnt\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.498629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.498199 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-metrics-certs\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.498629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.498283 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-maas-api-tls\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.498629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.498527 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ebc11-1e71-41c3-ab12-e33c21661cce-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.539198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.539171 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:10.599889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.599855 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-metrics-certs\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.599889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.599889 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-maas-api-tls\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.600098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.599994 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ebc11-1e71-41c3-ab12-e33c21661cce-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.600189 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.600174 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbnt\" (UniqueName: \"kubernetes.io/projected/dc1ebc11-1e71-41c3-ab12-e33c21661cce-kube-api-access-bjbnt\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.600816 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.600786 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ebc11-1e71-41c3-ab12-e33c21661cce-openshift-service-ca\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.603070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.603048 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-maas-api-tls\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.603222 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.603206 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-metrics-certs\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.611877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.611852 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbnt\" (UniqueName: \"kubernetes.io/projected/dc1ebc11-1e71-41c3-ab12-e33c21661cce-kube-api-access-bjbnt\") pod \"maas-api-e2e-api-b-b6b833-86764b4f4-x42mt\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.693739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.693122 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:10.766788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:10.764125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:11.231625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:11.231590 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt"] Oct 31 01:46:11.398101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:11.397994 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" event={"ID":"dc1ebc11-1e71-41c3-ab12-e33c21661cce","Type":"ContainerStarted","Data":"065c4bc5407a982010cf32ae1fe223a71c3df47aa036a230c4d32a70115e5088"} Oct 31 01:46:11.590312 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:11.590276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:12.406097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:12.406064 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" event={"ID":"dc1ebc11-1e71-41c3-ab12-e33c21661cce","Type":"ContainerStarted","Data":"5227bb76511d104852f24ff989364e00dc9daea22c9bbed533a5287bf07ee7de"} Oct 31 01:46:12.406323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:12.406244 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:12.427765 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:12.427464 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" podStartSLOduration=1.9302353970000001 podStartE2EDuration="2.427447087s" totalImagesPullingTime="497.21169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:12.426261433 +0000 UTC m=+1572.806506675" watchObservedRunningTime="2026-10-31 01:46:12.427447087 +0000 UTC m=+1572.807692312" Oct 31 01:46:12.509098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:12.509060 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:13.353737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:13.350885 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:14.212531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:14.212490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:14.395777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:14.395743 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:14.545262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:14.545224 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:15.027529 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:15.027495 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:15.678620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:15.678593 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:15.824530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:15.824496 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:15.994830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:15.991225 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:16.718412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:16.718364 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:17.629609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:17.629541 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:18.265852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:18.265814 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:18.433347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:18.433306 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:46:18.979278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:18.979246 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:19.080100 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.080065 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w"] Oct 31 01:46:19.084491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.084470 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.088131 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.088110 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1b634e-openshift-default-dockercfg-qdmlf\"" Oct 31 01:46:19.099162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.099135 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w"] Oct 31 01:46:19.191294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191265 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191461 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191304 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191461 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191331 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefa558f-1096-45be-8508-42c9d9d77186-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191494 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191533 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191589 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefa558f-1096-45be-8508-42c9d9d77186-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191704 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191691 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191745 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.191812 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.191766 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwl6\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-kube-api-access-hqwl6\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.292650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292622 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292659 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292679 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefa558f-1096-45be-8508-42c9d9d77186-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292744 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292772 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292818 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefa558f-1096-45be-8508-42c9d9d77186-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292936 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292967 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.292989 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwl6\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-kube-api-access-hqwl6\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.293078 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-certs\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.293101 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-data\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.293206 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-credential-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.293217 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-socket\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.293606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.293599 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefa558f-1096-45be-8508-42c9d9d77186-istiod-ca-cert\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.295404 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.295382 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-envoy\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.295653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.295635 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefa558f-1096-45be-8508-42c9d9d77186-istio-podinfo\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.304921 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.304900 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-istio-token\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.305008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.304899 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwl6\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-kube-api-access-hqwl6\") pod \"e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.400112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.400083 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:19.726842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.726753 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w"] Oct 31 01:46:19.736968 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:46:19.736932 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefa558f_1096_45be_8508_42c9d9d77186.slice/crio-52361c569c7bbe2fa27263cd16c5448f3d4e407ef486d26e7e33b0ebf02ff943 WatchSource:0}: Error finding container 52361c569c7bbe2fa27263cd16c5448f3d4e407ef486d26e7e33b0ebf02ff943: Status 404 returned error can't find the container with id 52361c569c7bbe2fa27263cd16c5448f3d4e407ef486d26e7e33b0ebf02ff943 Oct 31 01:46:19.748001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.747965 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:46:19.748196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.748177 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:46:19.748291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.748278 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:46:19.910744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:19.907762 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:20.477301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:20.477250 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" event={"ID":"aefa558f-1096-45be-8508-42c9d9d77186","Type":"ContainerStarted","Data":"81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72"} Oct 31 01:46:20.477869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:20.477322 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" event={"ID":"aefa558f-1096-45be-8508-42c9d9d77186","Type":"ContainerStarted","Data":"52361c569c7bbe2fa27263cd16c5448f3d4e407ef486d26e7e33b0ebf02ff943"} Oct 31 01:46:20.500500 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:20.500444 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podStartSLOduration=1.5004272090000002 podStartE2EDuration="1.500427209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:20.498953356 +0000 UTC m=+1580.879198571" watchObservedRunningTime="2026-10-31 01:46:20.500427209 +0000 UTC m=+1580.880672435" Oct 31 01:46:20.734524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:20.734438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:21.015555 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.015517 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq"] Oct 31 01:46:21.021206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.021180 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.028479 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.028452 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9cc66e-openshift-default-dockercfg-9r6q4\"" Oct 31 01:46:21.042608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.042581 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq"] Oct 31 01:46:21.154611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.154576 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.154846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.154623 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.154846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.154771 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.154959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.154880 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.154959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.154911 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.154959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.154940 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.154959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.154955 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.155152 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.155132 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.155223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.155208 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlrnh\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-kube-api-access-tlrnh\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.256740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.256693 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.256924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.256747 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.256924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.256801 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.256924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.256853 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlrnh\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-kube-api-access-tlrnh\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.256952 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.256984 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257042 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257089 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257108 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257364 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257160 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257535 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257491 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-credential-socket\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257640 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istiod-ca-cert\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.257932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257911 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-certs\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.258016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.257994 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-data\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.259297 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.259280 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-envoy\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.260251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.260232 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-podinfo\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.268457 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.268395 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-token\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.268703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.268679 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlrnh\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-kube-api-access-tlrnh\") pod \"e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.347466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.347434 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:21.361823 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.361796 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl"] Oct 31 01:46:21.419433 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.405969 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.422589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.420776 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:21.422589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.422465 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl"] Oct 31 01:46:21.425870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.425846 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0239db-serving-cert\"" Oct 31 01:46:21.430770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.426946 5090 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:46:21.430770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.427020 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 01:46:21.452698 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.452668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:21.463283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.463246 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k59b\" (UniqueName: \"kubernetes.io/projected/224a0ccc-a822-4e89-8117-4b122a04f686-kube-api-access-2k59b\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.463410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.463332 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-maas-api-tls\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.463410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.463392 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a0ccc-a822-4e89-8117-4b122a04f686-openshift-service-ca\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.463648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.463621 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-metrics-certs\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.615044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.572481 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-maas-api-tls\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.615044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.572542 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a0ccc-a822-4e89-8117-4b122a04f686-openshift-service-ca\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.615044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.572699 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-metrics-certs\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.615044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.572811 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k59b\" (UniqueName: \"kubernetes.io/projected/224a0ccc-a822-4e89-8117-4b122a04f686-kube-api-access-2k59b\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.615044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.598742 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a0ccc-a822-4e89-8117-4b122a04f686-openshift-service-ca\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.615044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.614426 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:21.629800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.626810 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-maas-api-tls\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.636853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.636823 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-metrics-certs\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.639101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.639056 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k59b\" (UniqueName: \"kubernetes.io/projected/224a0ccc-a822-4e89-8117-4b122a04f686-kube-api-access-2k59b\") pod \"maas-api-e2e-api-b-0239db-7d9669489d-mqthl\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.752783 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.752742 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:21.758867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.758844 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq"] Oct 31 01:46:21.763450 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:46:21.763427 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6fc01b_7f93_48ba_b3c1_452ae92d724b.slice/crio-2df219ff9184b0e09011d9aae565ead8fe9a64434f99a589dc45100d6327a8ba WatchSource:0}: Error finding container 2df219ff9184b0e09011d9aae565ead8fe9a64434f99a589dc45100d6327a8ba: Status 404 returned error can't find the container with id 2df219ff9184b0e09011d9aae565ead8fe9a64434f99a589dc45100d6327a8ba Oct 31 01:46:21.763696 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.763676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:21.766132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.766100 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:46:21.766223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.766175 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:46:21.766279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:21.766221 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:46:22.150654 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.150624 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl"] Oct 31 01:46:22.407819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.404124 5090 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:46:22.407819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.404186 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 01:46:22.571454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.567849 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" event={"ID":"ca6fc01b-7f93-48ba-b3c1-452ae92d724b","Type":"ContainerStarted","Data":"fff6118f8aff39b312eea08d69902a5918df4bdc3d0e9510a23ed9e132d5852d"} Oct 31 01:46:22.571454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.567913 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" event={"ID":"ca6fc01b-7f93-48ba-b3c1-452ae92d724b","Type":"ContainerStarted","Data":"2df219ff9184b0e09011d9aae565ead8fe9a64434f99a589dc45100d6327a8ba"} Oct 31 01:46:22.571454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.570919 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" event={"ID":"224a0ccc-a822-4e89-8117-4b122a04f686","Type":"ContainerStarted","Data":"f18ccd2249ec26fdb3416f70723d254bdb30f78e7a784077e8c7585c78acb746"} Oct 31 01:46:22.607214 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.607142 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podStartSLOduration=2.607122507 podStartE2EDuration="2.607122507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:22.60609063 +0000 UTC m=+1582.986335879" watchObservedRunningTime="2026-10-31 01:46:22.607122507 +0000 UTC m=+1582.987367731" Oct 31 01:46:22.712188 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:22.712144 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:23.348831 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.348798 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:23.350288 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.350266 5090 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 01:46:23.350421 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.350312 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 01:46:23.401740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.401164 5090 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:46:23.401740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.401231 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 01:46:23.628806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.628029 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" event={"ID":"224a0ccc-a822-4e89-8117-4b122a04f686","Type":"ContainerStarted","Data":"5919762ae3bfafddbe86d9c40ba26d9277de7431f6efb2ab935c0242acc72430"} Oct 31 01:46:23.628806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.628122 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:23.650730 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.650671 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" podStartSLOduration=2.074360041 podStartE2EDuration="2.650653361s" totalImagesPullingTime="576.29332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:23.646393994 +0000 UTC m=+1584.026639230" watchObservedRunningTime="2026-10-31 01:46:23.650653361 +0000 UTC m=+1584.030898602" Oct 31 01:46:23.769592 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:23.769562 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:24.134853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:24.134814 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:24.348155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:24.348122 5090 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 01:46:24.348331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:24.348183 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 01:46:24.420793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:24.410139 5090 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:46:24.420793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:24.410199 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 01:46:24.635195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:24.635161 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:25.348173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:25.348131 5090 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 01:46:25.348646 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:25.348189 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 01:46:25.414741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:25.414116 5090 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:46:25.414741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:25.414180 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 01:46:25.912743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:25.911128 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:26.348298 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:26.348264 5090 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 01:46:26.348806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:26.348319 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 01:46:26.403558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:26.403088 5090 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 01:46:26.403558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:26.403150 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 01:46:26.706797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:26.702671 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:27.356741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:27.356411 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:27.361748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:27.357733 5090 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 01:46:27.361748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:27.357791 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 01:46:27.417190 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:27.417158 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:27.418963 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:27.418922 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:27.419829 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:27.419806 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:46:28.175750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:28.168668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:28.348596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:28.348566 5090 patch_prober.go:29] interesting pod/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 01:46:28.348799 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:28.348617 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 01:46:28.851686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:28.851650 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:29.353373 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:29.353342 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:29.353875 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:29.353855 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:29.354672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:29.354651 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:46:29.459533 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:29.459494 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:30.253466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:30.253432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:30.670603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:30.670527 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:46:31.046076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:31.046026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:31.900551 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:31.896617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:32.570169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:32.570117 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:33.176142 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:33.176105 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:33.783734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:33.783684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:34.348441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:34.348354 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:34.492763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:34.492709 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:34.790349 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:46:34.790318 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:46:35.122841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:35.122699 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:35.276030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:35.275999 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:36.049477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:36.049415 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:36.714801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:36.714768 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:37.448394 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:37.448357 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:38.233380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:38.233345 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:39.067587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:39.067552 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:40.041825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:40.039687 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:40.885057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:40.885011 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:41.192016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:41.191931 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:41.359882 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:41.359844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:41.882048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:41.879490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:42.547873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:42.547842 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:43.288326 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:43.288295 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:43.894577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:43.894445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:44.570019 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:44.568559 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:45.260587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:45.260539 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:45.958579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:45.955945 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:46.592612 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:46.592554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:47.260821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:47.260786 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:47.541123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:47.541029 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:47.686368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:47.686334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:48.021213 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:48.021161 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:48.669455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:48.669390 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:49.351403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:49.351372 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:49.955167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:49.955128 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:50.184389 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:50.184357 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:50.655733 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:50.655666 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:51.316677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.313180 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:51.605556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.605466 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:51.759074 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.759042 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg"] Oct 31 01:46:51.759471 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.759280 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" podUID="8d5b46ee-30e5-4a3e-be9b-07322139e1a6" containerName="maas-api" containerID="cri-o://aba8d44f5dd1b2daaa5009d3c172060b65322920091021f406acf2b2b1c13dc5" gracePeriod=30 Oct 31 01:46:51.842073 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.842045 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:51.902112 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.902075 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11422-41c96c"] Oct 31 01:46:51.907399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.907372 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:46:51.916880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.912329 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmw2f\"" Oct 31 01:46:51.916880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.914258 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11422-41c96c"] Oct 31 01:46:51.933991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.933959 5090 generic.go:356] "Generic (PLEG): container finished" podID="8d5b46ee-30e5-4a3e-be9b-07322139e1a6" containerID="aba8d44f5dd1b2daaa5009d3c172060b65322920091021f406acf2b2b1c13dc5" exitCode=0 Oct 31 01:46:51.934145 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.934092 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" event={"ID":"8d5b46ee-30e5-4a3e-be9b-07322139e1a6","Type":"ContainerDied","Data":"aba8d44f5dd1b2daaa5009d3c172060b65322920091021f406acf2b2b1c13dc5"} Oct 31 01:46:51.986520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:51.986492 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:52.035415 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.034178 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmtz\" (UniqueName: \"kubernetes.io/projected/c9f22521-0d79-4013-b735-3b7ec1515952-kube-api-access-5mmtz\") pod \"test-curl-11422-41c96c\" (UID: \"c9f22521-0d79-4013-b735-3b7ec1515952\") " pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:46:52.151797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.146761 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmtz\" (UniqueName: \"kubernetes.io/projected/c9f22521-0d79-4013-b735-3b7ec1515952-kube-api-access-5mmtz\") pod \"test-curl-11422-41c96c\" (UID: \"c9f22521-0d79-4013-b735-3b7ec1515952\") " pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:46:52.180812 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.152536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:52.237171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.233585 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmtz\" (UniqueName: \"kubernetes.io/projected/c9f22521-0d79-4013-b735-3b7ec1515952-kube-api-access-5mmtz\") pod \"test-curl-11422-41c96c\" (UID: \"c9f22521-0d79-4013-b735-3b7ec1515952\") " pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:46:52.249984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.249946 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:46:52.349178 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.343501 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:52.414945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.413709 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:46:52.454699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.454323 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-metrics-certs\") pod \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " Oct 31 01:46:52.454936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.454915 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-openshift-service-ca\") pod \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " Oct 31 01:46:52.455893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.455487 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-openshift-service-ca" pod "8d5b46ee-30e5-4a3e-be9b-07322139e1a6" (UID: "8d5b46ee-30e5-4a3e-be9b-07322139e1a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:46:52.456306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.455704 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-maas-api-tls\") pod \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " Oct 31 01:46:52.456468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.456453 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-kube-api-access-kjpwq\" (UniqueName: \"kubernetes.io/projected/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-kube-api-access-kjpwq\") pod \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\" (UID: \"8d5b46ee-30e5-4a3e-be9b-07322139e1a6\") " Oct 31 01:46:52.458293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.458256 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:46:52.461937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.461906 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-kube-api-access-kjpwq" pod "8d5b46ee-30e5-4a3e-be9b-07322139e1a6" (UID: "8d5b46ee-30e5-4a3e-be9b-07322139e1a6"). InnerVolumeSpecName "kube-api-access-kjpwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:46:52.468561 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.467866 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-metrics-certs" pod "8d5b46ee-30e5-4a3e-be9b-07322139e1a6" (UID: "8d5b46ee-30e5-4a3e-be9b-07322139e1a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:52.469037 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.469010 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-maas-api-tls" pod "8d5b46ee-30e5-4a3e-be9b-07322139e1a6" (UID: "8d5b46ee-30e5-4a3e-be9b-07322139e1a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:46:52.560355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.560257 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:46:52.560355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.560297 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:46:52.560355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.560313 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjpwq\" (UniqueName: \"kubernetes.io/projected/8d5b46ee-30e5-4a3e-be9b-07322139e1a6-kube-api-access-kjpwq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:46:52.604776 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.602448 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:52.663409 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.663367 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11422-41c96c"] Oct 31 01:46:52.668267 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:46:52.668228 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9f22521_0d79_4013_b735_3b7ec1515952.slice/crio-1489f778b5ab1dd8cc63b1a5ab46c30886bb087ce4e5e0942fd465036d820928 WatchSource:0}: Error finding container 1489f778b5ab1dd8cc63b1a5ab46c30886bb087ce4e5e0942fd465036d820928: Status 404 returned error can't find the container with id 1489f778b5ab1dd8cc63b1a5ab46c30886bb087ce4e5e0942fd465036d820928 Oct 31 01:46:52.670652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.670616 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:46:52.945350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.945269 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" Oct 31 01:46:52.945350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.945297 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" event={"ID":"8d5b46ee-30e5-4a3e-be9b-07322139e1a6","Type":"ContainerDied","Data":"c6f4e840f88c6222dcc0fb94d29a92f2fe8a597f5d8e7e943389f77e3b0aa63b"} Oct 31 01:46:52.945859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.945363 5090 scope.go:122] "RemoveContainer" containerID="aba8d44f5dd1b2daaa5009d3c172060b65322920091021f406acf2b2b1c13dc5" Oct 31 01:46:52.951421 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.951366 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11422-41c96c" event={"ID":"c9f22521-0d79-4013-b735-3b7ec1515952","Type":"ContainerStarted","Data":"1489f778b5ab1dd8cc63b1a5ab46c30886bb087ce4e5e0942fd465036d820928"} Oct 31 01:46:52.971811 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.971785 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg"] Oct 31 01:46:52.979375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:52.979348 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg"] Oct 31 01:46:53.165061 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:53.165027 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:53.543178 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:53.543146 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:53.682065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:53.682024 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:53.753272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:53.753240 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5b46ee-30e5-4a3e-be9b-07322139e1a6" path="/var/lib/kubelet/pods/8d5b46ee-30e5-4a3e-be9b-07322139e1a6/volumes" Oct 31 01:46:53.975743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:53.975656 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:54.715137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:54.715027 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:55.234339 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:55.234293 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9cc66e-74d656df45-99czg" podUID="8d5b46ee-30e5-4a3e-be9b-07322139e1a6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.86:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:46:55.470909 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:55.470807 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:55.988896 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:55.988859 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11422-41c96c" event={"ID":"c9f22521-0d79-4013-b735-3b7ec1515952","Type":"ContainerStarted","Data":"7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f"} Oct 31 01:46:56.019970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:56.019919 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11422-41c96c" podStartSLOduration=2.662647613 podStartE2EDuration="5.019905339s" totalImagesPullingTime="2.357257726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:46:56.013975817 +0000 UTC m=+1616.394221051" watchObservedRunningTime="2026-10-31 01:46:56.019905339 +0000 UTC m=+1616.400150562" Oct 31 01:46:56.358738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:56.357935 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:56.764211 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:56.764180 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11422-41c96c"] Oct 31 01:46:56.769097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:56.769068 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11422-41c96c"] Oct 31 01:46:57.178180 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:57.178090 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:58.005546 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:58.005482 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11422-41c96c" podUID="c9f22521-0d79-4013-b735-3b7ec1515952" containerName="test-curl-11422-41c96c" containerID="cri-o://7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f" gracePeriod=2 Oct 31 01:46:58.048288 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:58.048251 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:58.579901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:58.579865 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:58.719578 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:58.719545 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:59.283478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:59.283396 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:59.474966 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:59.474936 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:59.632793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:59.632326 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:46:59.909097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:59.909023 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq"] Oct 31 01:46:59.909249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:59.909229 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" containerID="cri-o://fff6118f8aff39b312eea08d69902a5918df4bdc3d0e9510a23ed9e132d5852d" gracePeriod=30 Oct 31 01:46:59.995995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:46:59.990650 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:00.138741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.099005 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw"] Oct 31 01:47:00.149862 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.149823 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" podUID="505db6a7-31d6-49a7-892b-07cfc82f5caa" containerName="maas-api" containerID="cri-o://b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3" gracePeriod=30 Oct 31 01:47:00.164450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.164407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:00.431237 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.431077 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:00.594155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.594061 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:00.740153 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.740109 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:47:00.757730 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.757685 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:47:00.759171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.759143 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:00.857459 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.857426 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-metrics-certs\") pod \"505db6a7-31d6-49a7-892b-07cfc82f5caa\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " Oct 31 01:47:00.857650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.857585 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/505db6a7-31d6-49a7-892b-07cfc82f5caa-kube-api-access-tdj8d\" (UniqueName: \"kubernetes.io/projected/505db6a7-31d6-49a7-892b-07cfc82f5caa-kube-api-access-tdj8d\") pod \"505db6a7-31d6-49a7-892b-07cfc82f5caa\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " Oct 31 01:47:00.857650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.857628 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/505db6a7-31d6-49a7-892b-07cfc82f5caa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505db6a7-31d6-49a7-892b-07cfc82f5caa-openshift-service-ca\") pod \"505db6a7-31d6-49a7-892b-07cfc82f5caa\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " Oct 31 01:47:00.857650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.857648 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-maas-api-tls\") pod \"505db6a7-31d6-49a7-892b-07cfc82f5caa\" (UID: \"505db6a7-31d6-49a7-892b-07cfc82f5caa\") " Oct 31 01:47:00.857799 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.857693 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f22521-0d79-4013-b735-3b7ec1515952-kube-api-access-5mmtz\" (UniqueName: \"kubernetes.io/projected/c9f22521-0d79-4013-b735-3b7ec1515952-kube-api-access-5mmtz\") pod \"c9f22521-0d79-4013-b735-3b7ec1515952\" (UID: \"c9f22521-0d79-4013-b735-3b7ec1515952\") " Oct 31 01:47:00.858190 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.858130 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505db6a7-31d6-49a7-892b-07cfc82f5caa-openshift-service-ca" pod "505db6a7-31d6-49a7-892b-07cfc82f5caa" (UID: "505db6a7-31d6-49a7-892b-07cfc82f5caa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:00.858539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.858515 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/505db6a7-31d6-49a7-892b-07cfc82f5caa-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:00.859985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.859951 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-metrics-certs" pod "505db6a7-31d6-49a7-892b-07cfc82f5caa" (UID: "505db6a7-31d6-49a7-892b-07cfc82f5caa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:00.860586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.860562 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505db6a7-31d6-49a7-892b-07cfc82f5caa-kube-api-access-tdj8d" pod "505db6a7-31d6-49a7-892b-07cfc82f5caa" (UID: "505db6a7-31d6-49a7-892b-07cfc82f5caa"). InnerVolumeSpecName "kube-api-access-tdj8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:00.860684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.860622 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-maas-api-tls" pod "505db6a7-31d6-49a7-892b-07cfc82f5caa" (UID: "505db6a7-31d6-49a7-892b-07cfc82f5caa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:00.861048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.861031 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f22521-0d79-4013-b735-3b7ec1515952-kube-api-access-5mmtz" pod "c9f22521-0d79-4013-b735-3b7ec1515952" (UID: "c9f22521-0d79-4013-b735-3b7ec1515952"). InnerVolumeSpecName "kube-api-access-5mmtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:00.960217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.960137 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:00.960217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.960168 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdj8d\" (UniqueName: \"kubernetes.io/projected/505db6a7-31d6-49a7-892b-07cfc82f5caa-kube-api-access-tdj8d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:00.960217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.960180 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/505db6a7-31d6-49a7-892b-07cfc82f5caa-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:00.960217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:00.960191 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mmtz\" (UniqueName: \"kubernetes.io/projected/c9f22521-0d79-4013-b735-3b7ec1515952-kube-api-access-5mmtz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:01.130431 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.130400 5090 generic.go:356] "Generic (PLEG): container finished" podID="505db6a7-31d6-49a7-892b-07cfc82f5caa" containerID="b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3" exitCode=0 Oct 31 01:47:01.130626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.130453 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" Oct 31 01:47:01.130626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.130564 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" event={"ID":"505db6a7-31d6-49a7-892b-07cfc82f5caa","Type":"ContainerDied","Data":"b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3"} Oct 31 01:47:01.130626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.130615 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw" event={"ID":"505db6a7-31d6-49a7-892b-07cfc82f5caa","Type":"ContainerDied","Data":"1b878bcef8da2ea9d0c8ac6d36208e8b7021b56d0ddcb2c3e0f95cacef6d6da4"} Oct 31 01:47:01.130834 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.130648 5090 scope.go:122] "RemoveContainer" containerID="b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3" Oct 31 01:47:01.132903 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.132877 5090 generic.go:356] "Generic (PLEG): container finished" podID="c9f22521-0d79-4013-b735-3b7ec1515952" containerID="7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f" exitCode=137 Oct 31 01:47:01.133025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.132926 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:47:01.144838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.144681 5090 scope.go:122] "RemoveContainer" containerID="b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3" Oct 31 01:47:01.145264 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:01.145241 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3\": container with ID starting with b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3 not found: ID does not exist" containerID="b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3" Oct 31 01:47:01.145373 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.145276 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3"} err="failed to get container status \"b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3\": rpc error: code = NotFound desc = could not find container \"b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3\": container with ID starting with b8b7f64b0d2b8dabce988719511aa20e4fd8cf19cd7bacac47cb3fc67249efc3 not found: ID does not exist" Oct 31 01:47:01.145373 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.145299 5090 scope.go:122] "RemoveContainer" containerID="7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f" Oct 31 01:47:01.158129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.158106 5090 scope.go:122] "RemoveContainer" containerID="7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f" Oct 31 01:47:01.158379 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:01.158356 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f\": container with ID starting with 7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f not found: ID does not exist" containerID="7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f" Oct 31 01:47:01.158476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.158384 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f"} err="failed to get container status \"7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f\": rpc error: code = NotFound desc = could not find container \"7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f\": container with ID starting with 7505bf041d6968dcd069d24029014e5e632ff5977e23541ac9c7135d6cf7739f not found: ID does not exist" Oct 31 01:47:01.160602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.160576 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw"] Oct 31 01:47:01.162095 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:01.162069 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11422-41c96c\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="c9f22521-0d79-4013-b735-3b7ec1515952" pod="opendatahub/test-curl-11422-41c96c" Oct 31 01:47:01.166606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.166582 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1b634e-5dccbfd8d6-47tnw"] Oct 31 01:47:01.232151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.232051 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:01.463957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.463153 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:01.751677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.751647 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="505db6a7-31d6-49a7-892b-07cfc82f5caa" path="/var/lib/kubelet/pods/505db6a7-31d6-49a7-892b-07cfc82f5caa/volumes" Oct 31 01:47:01.752087 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:01.752074 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9f22521-0d79-4013-b735-3b7ec1515952" path="/var/lib/kubelet/pods/c9f22521-0d79-4013-b735-3b7ec1515952/volumes" Oct 31 01:47:02.164465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:02.164381 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:02.971465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:02.967925 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:03.633740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:03.632370 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:04.368528 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:04.368412 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:04.788940 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:04.788897 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:47:05.148111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.147953 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:05.177090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.177050 5090 generic.go:356] "Generic (PLEG): container finished" podID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerID="fff6118f8aff39b312eea08d69902a5918df4bdc3d0e9510a23ed9e132d5852d" exitCode=0 Oct 31 01:47:05.177280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.177121 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" event={"ID":"ca6fc01b-7f93-48ba-b3c1-452ae92d724b","Type":"ContainerDied","Data":"fff6118f8aff39b312eea08d69902a5918df4bdc3d0e9510a23ed9e132d5852d"} Oct 31 01:47:05.220848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.220820 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:47:05.365972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.365888 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-certs\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366040 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-credential-socket\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366081 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-podinfo\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366110 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-token\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-token\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366178 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-socket\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366191 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-certs" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:05.366313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366267 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-envoy\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366342 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istiod-ca-cert\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366376 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-data\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.366462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366450 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-credential-socket" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:05.366623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366485 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-kube-api-access-tlrnh\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-kube-api-access-tlrnh\") pod \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\" (UID: \"ca6fc01b-7f93-48ba-b3c1-452ae92d724b\") " Oct 31 01:47:05.367247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.366889 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-socket" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:05.367247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.367139 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istiod-ca-cert" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:05.367247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.367173 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.367247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.367199 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.367247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.367214 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.367618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.367422 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-data" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:05.369683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.369658 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-podinfo" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:47:05.370125 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.369771 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-kube-api-access-tlrnh" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "kube-api-access-tlrnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:05.370125 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.369701 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-envoy" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:05.370125 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.370107 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-token" pod "ca6fc01b-7f93-48ba-b3c1-452ae92d724b" (UID: "ca6fc01b-7f93-48ba-b3c1-452ae92d724b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:05.502280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.476789 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.502280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.476834 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.502280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.476848 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.502280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.476862 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.502280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.476875 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.502280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.476889 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlrnh\" (UniqueName: \"kubernetes.io/projected/ca6fc01b-7f93-48ba-b3c1-452ae92d724b-kube-api-access-tlrnh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:05.857743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:05.857621 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:06.185170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:06.185094 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" Oct 31 01:47:06.185170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:06.185108 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq" event={"ID":"ca6fc01b-7f93-48ba-b3c1-452ae92d724b","Type":"ContainerDied","Data":"2df219ff9184b0e09011d9aae565ead8fe9a64434f99a589dc45100d6327a8ba"} Oct 31 01:47:06.185170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:06.185163 5090 scope.go:122] "RemoveContainer" containerID="fff6118f8aff39b312eea08d69902a5918df4bdc3d0e9510a23ed9e132d5852d" Oct 31 01:47:06.203612 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:06.203586 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq"] Oct 31 01:47:06.209424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:06.209403 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9cc66e-openshift-default-7757695b65-6tqdq"] Oct 31 01:47:06.537314 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:06.537283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:07.247353 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:07.247322 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:07.712911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:07.712820 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:07.755147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:07.755113 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" path="/var/lib/kubelet/pods/ca6fc01b-7f93-48ba-b3c1-452ae92d724b/volumes" Oct 31 01:47:07.860395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:07.860359 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:08.008235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:08.008204 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:08.226007 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:08.225968 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w"] Oct 31 01:47:08.226702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:08.226673 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" containerID="cri-o://81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72" gracePeriod=30 Oct 31 01:47:08.312640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:08.312538 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:08.689417 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:08.689314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:09.624868 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:09.624835 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:10.267946 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:10.267913 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:11.024090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:11.024061 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:11.163871 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:11.163836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:11.825132 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:11.825096 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:12.421491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:12.421458 5090 patch_prober.go:29] interesting pod/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 01:47:12.421978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:12.421528 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 01:47:12.478317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:12.478232 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:13.161438 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.161322 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:13.526990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.526967 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:47:13.604736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.604689 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-envoy\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.604924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.604783 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-socket\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.604924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.604849 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aefa558f-1096-45be-8508-42c9d9d77186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefa558f-1096-45be-8508-42c9d9d77186-istiod-ca-cert\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.604924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.604878 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aefa558f-1096-45be-8508-42c9d9d77186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefa558f-1096-45be-8508-42c9d9d77186-istio-podinfo\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.606306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.604943 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-kube-api-access-hqwl6\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-kube-api-access-hqwl6\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.606306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.604978 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-certs\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.606306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.605022 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-credential-socket\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.606306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.605093 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-socket" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:13.606306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.605169 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-data\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.606306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.605238 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-istio-token\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-istio-token\") pod \"aefa558f-1096-45be-8508-42c9d9d77186\" (UID: \"aefa558f-1096-45be-8508-42c9d9d77186\") " Oct 31 01:47:13.606306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.605325 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefa558f-1096-45be-8508-42c9d9d77186-istiod-ca-cert" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:13.607375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.605990 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.607375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.606734 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefa558f-1096-45be-8508-42c9d9d77186-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.607375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.607016 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-credential-socket" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:13.607375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.607220 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-certs" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:13.607749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.607662 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-data" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:13.608058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.608033 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-kube-api-access-hqwl6" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "kube-api-access-hqwl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:13.608694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.608665 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aefa558f-1096-45be-8508-42c9d9d77186-istio-podinfo" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:47:13.608835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.608799 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-istio-token" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:13.608947 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.608927 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-envoy" pod "aefa558f-1096-45be-8508-42c9d9d77186" (UID: "aefa558f-1096-45be-8508-42c9d9d77186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:47:13.708869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.708792 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.708869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.708829 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.708869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.708844 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.708869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.708859 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.709098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.708872 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefa558f-1096-45be-8508-42c9d9d77186-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.709098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.708886 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwl6\" (UniqueName: \"kubernetes.io/projected/aefa558f-1096-45be-8508-42c9d9d77186-kube-api-access-hqwl6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.709098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.708901 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefa558f-1096-45be-8508-42c9d9d77186-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:13.796078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:13.796052 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:14.276257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.276224 5090 generic.go:356] "Generic (PLEG): container finished" podID="aefa558f-1096-45be-8508-42c9d9d77186" containerID="81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72" exitCode=0 Oct 31 01:47:14.276482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.276273 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" Oct 31 01:47:14.276482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.276312 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" event={"ID":"aefa558f-1096-45be-8508-42c9d9d77186","Type":"ContainerDied","Data":"81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72"} Oct 31 01:47:14.276482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.276380 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w" event={"ID":"aefa558f-1096-45be-8508-42c9d9d77186","Type":"ContainerDied","Data":"52361c569c7bbe2fa27263cd16c5448f3d4e407ef486d26e7e33b0ebf02ff943"} Oct 31 01:47:14.276482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.276408 5090 scope.go:122] "RemoveContainer" containerID="81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72" Oct 31 01:47:14.286753 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.286702 5090 scope.go:122] "RemoveContainer" containerID="81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72" Oct 31 01:47:14.287033 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:14.287016 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72\": container with ID starting with 81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72 not found: ID does not exist" containerID="81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72" Oct 31 01:47:14.287109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.287051 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72"} err="failed to get container status \"81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72\": rpc error: code = NotFound desc = could not find container \"81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72\": container with ID starting with 81922a91b2ce3cee1e146d4993e084cc8a72bdba41f6d981c3c6a77f4dca5b72 not found: ID does not exist" Oct 31 01:47:14.295436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.295413 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w"] Oct 31 01:47:14.299765 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.299744 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1b634e-openshift-default-d66ff8bf5-p976w"] Oct 31 01:47:14.513660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:14.513540 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:15.365926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:15.365894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:15.747030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:15.747001 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aefa558f-1096-45be-8508-42c9d9d77186" path="/var/lib/kubelet/pods/aefa558f-1096-45be-8508-42c9d9d77186/volumes" Oct 31 01:47:16.071091 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:16.070984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:16.711619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:16.711584 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:17.351788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:17.350620 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:17.982569 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:17.982530 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:18.701544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:18.701511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:19.045509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:19.045477 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:19.207223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:19.207190 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:19.530036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:19.529998 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:19.674832 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:19.674796 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:19.995886 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:19.995858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:20.162806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:20.162654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:20.526403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:20.526358 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:21.185424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:21.185377 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:21.782143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:21.782112 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt"] Oct 31 01:47:21.782644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:21.782611 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" podUID="dc1ebc11-1e71-41c3-ab12-e33c21661cce" containerName="maas-api" containerID="cri-o://5227bb76511d104852f24ff989364e00dc9daea22c9bbed533a5287bf07ee7de" gracePeriod=30 Oct 31 01:47:21.837499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:21.837458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:21.999789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:21.997137 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:22.290740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.279759 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:22.371829 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.371759 5090 generic.go:356] "Generic (PLEG): container finished" podID="dc1ebc11-1e71-41c3-ab12-e33c21661cce" containerID="5227bb76511d104852f24ff989364e00dc9daea22c9bbed533a5287bf07ee7de" exitCode=0 Oct 31 01:47:22.372102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.372085 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" event={"ID":"dc1ebc11-1e71-41c3-ab12-e33c21661cce","Type":"ContainerDied","Data":"5227bb76511d104852f24ff989364e00dc9daea22c9bbed533a5287bf07ee7de"} Oct 31 01:47:22.372237 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.372225 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" event={"ID":"dc1ebc11-1e71-41c3-ab12-e33c21661cce","Type":"ContainerDied","Data":"065c4bc5407a982010cf32ae1fe223a71c3df47aa036a230c4d32a70115e5088"} Oct 31 01:47:22.372314 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.372305 5090 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="065c4bc5407a982010cf32ae1fe223a71c3df47aa036a230c4d32a70115e5088" Oct 31 01:47:22.398604 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.398569 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:47:22.507339 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.507307 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:22.511256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.511222 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-metrics-certs\") pod \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " Oct 31 01:47:22.511256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.511268 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1ebc11-1e71-41c3-ab12-e33c21661cce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ebc11-1e71-41c3-ab12-e33c21661cce-openshift-service-ca\") pod \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " Oct 31 01:47:22.511503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.511353 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-maas-api-tls\") pod \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " Oct 31 01:47:22.511555 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.511502 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1ebc11-1e71-41c3-ab12-e33c21661cce-kube-api-access-bjbnt\" (UniqueName: \"kubernetes.io/projected/dc1ebc11-1e71-41c3-ab12-e33c21661cce-kube-api-access-bjbnt\") pod \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\" (UID: \"dc1ebc11-1e71-41c3-ab12-e33c21661cce\") " Oct 31 01:47:22.511704 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.511676 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1ebc11-1e71-41c3-ab12-e33c21661cce-openshift-service-ca" pod "dc1ebc11-1e71-41c3-ab12-e33c21661cce" (UID: "dc1ebc11-1e71-41c3-ab12-e33c21661cce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:22.512129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.512100 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc1ebc11-1e71-41c3-ab12-e33c21661cce-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:22.514886 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.514828 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-metrics-certs" pod "dc1ebc11-1e71-41c3-ab12-e33c21661cce" (UID: "dc1ebc11-1e71-41c3-ab12-e33c21661cce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:22.515012 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.514989 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-maas-api-tls" pod "dc1ebc11-1e71-41c3-ab12-e33c21661cce" (UID: "dc1ebc11-1e71-41c3-ab12-e33c21661cce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:22.519345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.519322 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1ebc11-1e71-41c3-ab12-e33c21661cce-kube-api-access-bjbnt" pod "dc1ebc11-1e71-41c3-ab12-e33c21661cce" (UID: "dc1ebc11-1e71-41c3-ab12-e33c21661cce"). InnerVolumeSpecName "kube-api-access-bjbnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:22.613103 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.613052 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:22.613103 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.613097 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbnt\" (UniqueName: \"kubernetes.io/projected/dc1ebc11-1e71-41c3-ab12-e33c21661cce-kube-api-access-bjbnt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:22.613375 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.613112 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc1ebc11-1e71-41c3-ab12-e33c21661cce-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:22.665667 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:22.665568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:23.380759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:23.380729 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt" Oct 31 01:47:23.415124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:23.415079 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt"] Oct 31 01:47:23.420234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:23.420206 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b6b833-86764b4f4-x42mt"] Oct 31 01:47:23.474738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:23.469643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:23.752648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:23.752612 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1ebc11-1e71-41c3-ab12-e33c21661cce" path="/var/lib/kubelet/pods/dc1ebc11-1e71-41c3-ab12-e33c21661cce/volumes" Oct 31 01:47:24.012044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:24.011952 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:24.152670 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:24.152634 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:24.560250 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:24.560208 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:25.505742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:25.484529 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j"] Oct 31 01:47:25.506260 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:25.506225 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" podUID="ede888a8-8881-464a-8706-972e92fa52ba" containerName="maas-api" containerID="cri-o://3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876" gracePeriod=30 Oct 31 01:47:25.633696 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:25.633664 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:25.859399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:25.856887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:26.393532 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.393494 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:47:26.429090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.429058 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:26.478564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.478475 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-metrics-certs\") pod \"ede888a8-8881-464a-8706-972e92fa52ba\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " Oct 31 01:47:26.478564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.478527 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede888a8-8881-464a-8706-972e92fa52ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede888a8-8881-464a-8706-972e92fa52ba-openshift-service-ca\") pod \"ede888a8-8881-464a-8706-972e92fa52ba\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " Oct 31 01:47:26.478853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.478586 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede888a8-8881-464a-8706-972e92fa52ba-kube-api-access-2qjbb\" (UniqueName: \"kubernetes.io/projected/ede888a8-8881-464a-8706-972e92fa52ba-kube-api-access-2qjbb\") pod \"ede888a8-8881-464a-8706-972e92fa52ba\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " Oct 31 01:47:26.478954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.478617 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-maas-api-tls\") pod \"ede888a8-8881-464a-8706-972e92fa52ba\" (UID: \"ede888a8-8881-464a-8706-972e92fa52ba\") " Oct 31 01:47:26.479049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.479016 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede888a8-8881-464a-8706-972e92fa52ba-openshift-service-ca" pod "ede888a8-8881-464a-8706-972e92fa52ba" (UID: "ede888a8-8881-464a-8706-972e92fa52ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:26.479978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.479939 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede888a8-8881-464a-8706-972e92fa52ba-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:26.481057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.481022 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-metrics-certs" pod "ede888a8-8881-464a-8706-972e92fa52ba" (UID: "ede888a8-8881-464a-8706-972e92fa52ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:26.481057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.481049 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-maas-api-tls" pod "ede888a8-8881-464a-8706-972e92fa52ba" (UID: "ede888a8-8881-464a-8706-972e92fa52ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:26.481185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.481028 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede888a8-8881-464a-8706-972e92fa52ba-kube-api-access-2qjbb" pod "ede888a8-8881-464a-8706-972e92fa52ba" (UID: "ede888a8-8881-464a-8706-972e92fa52ba"). InnerVolumeSpecName "kube-api-access-2qjbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:26.581503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.581467 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:26.581503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.581504 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qjbb\" (UniqueName: \"kubernetes.io/projected/ede888a8-8881-464a-8706-972e92fa52ba-kube-api-access-2qjbb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:26.581785 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.581520 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede888a8-8881-464a-8706-972e92fa52ba-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:26.668764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.668733 5090 generic.go:356] "Generic (PLEG): container finished" podID="ede888a8-8881-464a-8706-972e92fa52ba" containerID="3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876" exitCode=0 Oct 31 01:47:26.669278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.668785 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" Oct 31 01:47:26.669278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.668816 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" event={"ID":"ede888a8-8881-464a-8706-972e92fa52ba","Type":"ContainerDied","Data":"3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876"} Oct 31 01:47:26.669278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.668880 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j" event={"ID":"ede888a8-8881-464a-8706-972e92fa52ba","Type":"ContainerDied","Data":"7b561864283afd6e2f2248aa5719a0406348ed05355354ac8ae66b9e07f91034"} Oct 31 01:47:26.669278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.668963 5090 scope.go:122] "RemoveContainer" containerID="3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876" Oct 31 01:47:26.682591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.682563 5090 scope.go:122] "RemoveContainer" containerID="3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876" Oct 31 01:47:26.682945 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:26.682920 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876\": container with ID starting with 3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876 not found: ID does not exist" containerID="3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876" Oct 31 01:47:26.683045 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.682958 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876"} err="failed to get container status \"3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876\": rpc error: code = NotFound desc = could not find container \"3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876\": container with ID starting with 3698e31c8f4d2903963a9e2116fc8c9ff492829ceedd84c26e6dc55540ca2876 not found: ID does not exist" Oct 31 01:47:26.700823 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.700794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:26.703911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.703882 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j"] Oct 31 01:47:26.707362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:26.707342 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-thk7j"] Oct 31 01:47:27.505565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:27.505533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:27.747256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:27.747213 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede888a8-8881-464a-8706-972e92fa52ba" path="/var/lib/kubelet/pods/ede888a8-8881-464a-8706-972e92fa52ba/volumes" Oct 31 01:47:28.190332 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:28.190277 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:28.849187 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:28.849124 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:28.996405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:28.996360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:29.479156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.479121 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss"] Oct 31 01:47:29.480129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480106 5090 cpu_manager.go:414] "containerMap: removing container" podUID="505db6a7-31d6-49a7-892b-07cfc82f5caa" containerName="maas-api" Oct 31 01:47:29.480129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480128 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="505db6a7-31d6-49a7-892b-07cfc82f5caa" containerName="maas-api" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480152 5090 cpu_manager.go:414] "containerMap: removing container" podUID="8d5b46ee-30e5-4a3e-be9b-07322139e1a6" containerName="maas-api" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480161 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5b46ee-30e5-4a3e-be9b-07322139e1a6" containerName="maas-api" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480170 5090 cpu_manager.go:414] "containerMap: removing container" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480178 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480192 5090 cpu_manager.go:414] "containerMap: removing container" podUID="dc1ebc11-1e71-41c3-ab12-e33c21661cce" containerName="maas-api" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480198 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1ebc11-1e71-41c3-ab12-e33c21661cce" containerName="maas-api" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480221 5090 cpu_manager.go:414] "containerMap: removing container" podUID="ede888a8-8881-464a-8706-972e92fa52ba" containerName="maas-api" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480228 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede888a8-8881-464a-8706-972e92fa52ba" containerName="maas-api" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480243 5090 cpu_manager.go:414] "containerMap: removing container" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480250 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480282 5090 cpu_manager.go:414] "containerMap: removing container" podUID="c9f22521-0d79-4013-b735-3b7ec1515952" containerName="test-curl-11422-41c96c" Oct 31 01:47:29.480342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480289 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f22521-0d79-4013-b735-3b7ec1515952" containerName="test-curl-11422-41c96c" Oct 31 01:47:29.480898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480424 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1ebc11-1e71-41c3-ab12-e33c21661cce" containerName="maas-api" Oct 31 01:47:29.480898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480437 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5b46ee-30e5-4a3e-be9b-07322139e1a6" containerName="maas-api" Oct 31 01:47:29.480898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480447 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca6fc01b-7f93-48ba-b3c1-452ae92d724b" containerName="istio-proxy" Oct 31 01:47:29.480898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480459 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede888a8-8881-464a-8706-972e92fa52ba" containerName="maas-api" Oct 31 01:47:29.480898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480470 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefa558f-1096-45be-8508-42c9d9d77186" containerName="istio-proxy" Oct 31 01:47:29.480898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480481 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f22521-0d79-4013-b735-3b7ec1515952" containerName="test-curl-11422-41c96c" Oct 31 01:47:29.480898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.480490 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="505db6a7-31d6-49a7-892b-07cfc82f5caa" containerName="maas-api" Oct 31 01:47:29.489659 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.489633 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.499678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.499651 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-50ac12-serving-cert\"" Oct 31 01:47:29.504429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.504399 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss"] Oct 31 01:47:29.533120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.533093 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gf97\" (UniqueName: \"kubernetes.io/projected/899abbef-2312-483f-b16c-8cc2728a31e2-kube-api-access-2gf97\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.533357 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.533329 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-maas-api-tls\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.533462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.533382 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/899abbef-2312-483f-b16c-8cc2728a31e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.533527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.533513 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-metrics-certs\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.566030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.565993 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:29.634870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.634833 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gf97\" (UniqueName: \"kubernetes.io/projected/899abbef-2312-483f-b16c-8cc2728a31e2-kube-api-access-2gf97\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.635034 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.634901 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-maas-api-tls\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.635088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.635058 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/899abbef-2312-483f-b16c-8cc2728a31e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.635155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.635141 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-metrics-certs\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.635824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.635797 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/899abbef-2312-483f-b16c-8cc2728a31e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.637548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.637521 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-maas-api-tls\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.637685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.637667 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-metrics-certs\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.646118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.646097 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gf97\" (UniqueName: \"kubernetes.io/projected/899abbef-2312-483f-b16c-8cc2728a31e2-kube-api-access-2gf97\") pod \"maas-api-e2e-api-a-50ac12-cb4568b79-l7bss\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.701023 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.700993 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:29.819708 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.819670 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:29.838326 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.838304 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:29.994052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:29.990434 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:30.125792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.112787 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss"] Oct 31 01:47:30.224740 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:47:30.224082 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899abbef_2312_483f_b16c_8cc2728a31e2.slice/crio-425884615b9a8072a5a3a3bec967edb1b0080888c9eb8562a2ef2e45691441ee WatchSource:0}: Error finding container 425884615b9a8072a5a3a3bec967edb1b0080888c9eb8562a2ef2e45691441ee: Status 404 returned error can't find the container with id 425884615b9a8072a5a3a3bec967edb1b0080888c9eb8562a2ef2e45691441ee Oct 31 01:47:30.309545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.309492 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl"] Oct 31 01:47:30.324340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.317500 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" podUID="224a0ccc-a822-4e89-8117-4b122a04f686" containerName="maas-api" containerID="cri-o://5919762ae3bfafddbe86d9c40ba26d9277de7431f6efb2ab935c0242acc72430" gracePeriod=30 Oct 31 01:47:30.393104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.393024 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:30.586741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.582281 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:30.710705 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.708674 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" podUID="224a0ccc-a822-4e89-8117-4b122a04f686" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.91:8443/health\": dial tcp 10.132.0.91:8443: connect: connection refused" Oct 31 01:47:30.762031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.761857 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:30.770222 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.770036 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" event={"ID":"899abbef-2312-483f-b16c-8cc2728a31e2","Type":"ContainerStarted","Data":"425884615b9a8072a5a3a3bec967edb1b0080888c9eb8562a2ef2e45691441ee"} Oct 31 01:47:30.851750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.851570 5090 generic.go:356] "Generic (PLEG): container finished" podID="224a0ccc-a822-4e89-8117-4b122a04f686" containerID="5919762ae3bfafddbe86d9c40ba26d9277de7431f6efb2ab935c0242acc72430" exitCode=0 Oct 31 01:47:30.851750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.851680 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" event={"ID":"224a0ccc-a822-4e89-8117-4b122a04f686","Type":"ContainerDied","Data":"5919762ae3bfafddbe86d9c40ba26d9277de7431f6efb2ab935c0242acc72430"} Oct 31 01:47:30.920977 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.915457 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:30.939898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:30.939872 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:47:31.027509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.027475 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-metrics-certs\") pod \"224a0ccc-a822-4e89-8117-4b122a04f686\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " Oct 31 01:47:31.027964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.027551 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-maas-api-tls\") pod \"224a0ccc-a822-4e89-8117-4b122a04f686\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " Oct 31 01:47:31.027964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.027586 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/224a0ccc-a822-4e89-8117-4b122a04f686-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a0ccc-a822-4e89-8117-4b122a04f686-openshift-service-ca\") pod \"224a0ccc-a822-4e89-8117-4b122a04f686\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " Oct 31 01:47:31.027964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.027765 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/224a0ccc-a822-4e89-8117-4b122a04f686-kube-api-access-2k59b\" (UniqueName: \"kubernetes.io/projected/224a0ccc-a822-4e89-8117-4b122a04f686-kube-api-access-2k59b\") pod \"224a0ccc-a822-4e89-8117-4b122a04f686\" (UID: \"224a0ccc-a822-4e89-8117-4b122a04f686\") " Oct 31 01:47:31.028143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.028038 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224a0ccc-a822-4e89-8117-4b122a04f686-openshift-service-ca" pod "224a0ccc-a822-4e89-8117-4b122a04f686" (UID: "224a0ccc-a822-4e89-8117-4b122a04f686"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:47:31.028802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.028419 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/224a0ccc-a822-4e89-8117-4b122a04f686-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:31.030199 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.030175 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-metrics-certs" pod "224a0ccc-a822-4e89-8117-4b122a04f686" (UID: "224a0ccc-a822-4e89-8117-4b122a04f686"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:31.030474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.030441 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-maas-api-tls" pod "224a0ccc-a822-4e89-8117-4b122a04f686" (UID: "224a0ccc-a822-4e89-8117-4b122a04f686"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:47:31.031399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.031376 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224a0ccc-a822-4e89-8117-4b122a04f686-kube-api-access-2k59b" pod "224a0ccc-a822-4e89-8117-4b122a04f686" (UID: "224a0ccc-a822-4e89-8117-4b122a04f686"). InnerVolumeSpecName "kube-api-access-2k59b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:47:31.052829 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.052805 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:31.134800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.132873 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:31.134800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.132900 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/224a0ccc-a822-4e89-8117-4b122a04f686-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:31.134800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.132910 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k59b\" (UniqueName: \"kubernetes.io/projected/224a0ccc-a822-4e89-8117-4b122a04f686-kube-api-access-2k59b\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:47:31.537404 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.537363 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:31.866836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.866703 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" event={"ID":"899abbef-2312-483f-b16c-8cc2728a31e2","Type":"ContainerStarted","Data":"328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b"} Oct 31 01:47:31.867245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.867015 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:31.868628 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.868593 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" event={"ID":"224a0ccc-a822-4e89-8117-4b122a04f686","Type":"ContainerDied","Data":"f18ccd2249ec26fdb3416f70723d254bdb30f78e7a784077e8c7585c78acb746"} Oct 31 01:47:31.868781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.868679 5090 scope.go:122] "RemoveContainer" containerID="5919762ae3bfafddbe86d9c40ba26d9277de7431f6efb2ab935c0242acc72430" Oct 31 01:47:31.868781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.868607 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl" Oct 31 01:47:31.888418 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.888367 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" podStartSLOduration=2.189982247 podStartE2EDuration="2.888351645s" totalImagesPullingTime="698.369398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:31.885624381 +0000 UTC m=+1652.265869651" watchObservedRunningTime="2026-10-31 01:47:31.888351645 +0000 UTC m=+1652.268596870" Oct 31 01:47:31.901137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.901114 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl"] Oct 31 01:47:31.908513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:31.908485 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0239db-7d9669489d-mqthl"] Oct 31 01:47:32.037869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:32.037837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:32.816554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:32.816521 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:33.459221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:33.458659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:33.747105 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:33.747077 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="224a0ccc-a822-4e89-8117-4b122a04f686" path="/var/lib/kubelet/pods/224a0ccc-a822-4e89-8117-4b122a04f686/volumes" Oct 31 01:47:34.170283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:34.170176 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:34.789932 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:34.789901 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:47:34.829443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:34.829407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:35.662006 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:35.661164 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:36.358740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:36.347011 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:37.045095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:37.040817 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:37.406016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:37.405929 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:37.553963 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:37.553925 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:37.776709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:37.776675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:37.889187 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:37.889158 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:47:38.033790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:38.033086 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:38.432418 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:38.432339 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:38.570968 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:38.570933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:38.763926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:38.763898 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:38.991824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:38.990391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:39.143393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:39.143317 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:39.844225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:39.844186 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:40.535741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:40.535691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:41.145525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:41.145438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:41.988876 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:41.988845 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:42.752788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:42.746639 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:43.398753 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.398705 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj"] Oct 31 01:47:43.399336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.399321 5090 cpu_manager.go:414] "containerMap: removing container" podUID="224a0ccc-a822-4e89-8117-4b122a04f686" containerName="maas-api" Oct 31 01:47:43.399388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.399338 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="224a0ccc-a822-4e89-8117-4b122a04f686" containerName="maas-api" Oct 31 01:47:43.399448 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.399439 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="224a0ccc-a822-4e89-8117-4b122a04f686" containerName="maas-api" Oct 31 01:47:43.405766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.405738 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.414579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.414553 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cb5756cb-serving-cert\"" Oct 31 01:47:43.424450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.424419 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj"] Oct 31 01:47:43.452444 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.452406 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4xq\" (UniqueName: \"kubernetes.io/projected/8430f292-950d-4f71-b3b0-490829b0f473-kube-api-access-jm4xq\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.452618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.452466 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.452618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.452601 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8430f292-950d-4f71-b3b0-490829b0f473-openshift-service-ca\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.452785 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.452697 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-metrics-certs\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.501407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.501376 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:43.567739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.562783 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4xq\" (UniqueName: \"kubernetes.io/projected/8430f292-950d-4f71-b3b0-490829b0f473-kube-api-access-jm4xq\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.567739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.562874 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.567739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.562936 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8430f292-950d-4f71-b3b0-490829b0f473-openshift-service-ca\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.567739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.563039 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-metrics-certs\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.567739 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:43.564014 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-serving-cert: secret "maas-api-e2e-mt-cb5756cb-serving-cert" not found Oct 31 01:47:43.567739 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:47:43.564104 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls podName:8430f292-950d-4f71-b3b0-490829b0f473 nodeName:}" failed. No retries permitted until 2026-10-31 01:47:44.064082177 +0000 UTC m=+1664.444327402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls") pod "maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" (UID: "8430f292-950d-4f71-b3b0-490829b0f473") : secret "maas-api-e2e-mt-cb5756cb-serving-cert" not found Oct 31 01:47:43.567739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.565511 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8430f292-950d-4f71-b3b0-490829b0f473-openshift-service-ca\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.568705 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.568640 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-metrics-certs\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.589663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.589627 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4xq\" (UniqueName: \"kubernetes.io/projected/8430f292-950d-4f71-b3b0-490829b0f473-kube-api-access-jm4xq\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:43.642327 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.642290 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:43.786352 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:43.786314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:44.075807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:44.075563 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:44.089556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:44.089528 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls\") pod \"maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:44.129991 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:44.129960 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:44.327019 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:44.326927 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:44.354147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:44.336477 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:44.654785 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:44.654760 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj"] Oct 31 01:47:44.656076 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:47:44.656044 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8430f292_950d_4f71_b3b0_490829b0f473.slice/crio-8628bc5c36fec6c7e2f712df52a6853123e3a365fa844943a58066c9474005c5 WatchSource:0}: Error finding container 8628bc5c36fec6c7e2f712df52a6853123e3a365fa844943a58066c9474005c5: Status 404 returned error can't find the container with id 8628bc5c36fec6c7e2f712df52a6853123e3a365fa844943a58066c9474005c5 Oct 31 01:47:44.872824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:44.872799 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:45.020744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:45.018322 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:45.062495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:45.062454 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" event={"ID":"8430f292-950d-4f71-b3b0-490829b0f473","Type":"ContainerStarted","Data":"8628bc5c36fec6c7e2f712df52a6853123e3a365fa844943a58066c9474005c5"} Oct 31 01:47:45.371826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:45.366286 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:45.620594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:45.620498 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:45.892795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:45.878775 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:46.073560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:46.073522 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" event={"ID":"8430f292-950d-4f71-b3b0-490829b0f473","Type":"ContainerStarted","Data":"2bbb50ef467768b9e75b71ee74dfe5311eba31510c006fe59400527e0b33edb5"} Oct 31 01:47:46.073774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:46.073644 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:46.093554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:46.093498 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" podStartSLOduration=2.565433283 podStartE2EDuration="3.093481761s" totalImagesPullingTime="528.048478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:46.092373676 +0000 UTC m=+1666.472618900" watchObservedRunningTime="2026-10-31 01:47:46.093481761 +0000 UTC m=+1666.473726983" Oct 31 01:47:46.646467 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:46.646434 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:46.942581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:46.942483 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:47.290361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:47.290308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:48.023531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:48.023494 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:49.036014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:49.035988 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:49.195956 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:49.195923 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:50.157076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:50.157010 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:50.355393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:50.355341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:50.556827 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:50.556791 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:51.227743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:51.203736 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:52.118947 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:52.118919 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:47:52.355092 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:52.355057 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:53.054599 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:53.054539 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:53.761752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:53.759643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:54.438497 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:54.438442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:55.058518 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.058485 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt"] Oct 31 01:47:55.062939 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.062921 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.069236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.069213 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ea6cc6-serving-cert\"" Oct 31 01:47:55.074351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.074328 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt"] Oct 31 01:47:55.202068 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.202039 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-metrics-certs\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.202276 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.202207 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-maas-api-tls\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.202356 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.202342 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b1b214-72d8-4350-b6f4-bd7026b12a76-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.202420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.202365 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:55.202420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.202379 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmsj9\" (UniqueName: \"kubernetes.io/projected/87b1b214-72d8-4350-b6f4-bd7026b12a76-kube-api-access-tmsj9\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.315825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.315689 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-maas-api-tls\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.316295 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.316266 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b1b214-72d8-4350-b6f4-bd7026b12a76-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.317199 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.317174 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmsj9\" (UniqueName: \"kubernetes.io/projected/87b1b214-72d8-4350-b6f4-bd7026b12a76-kube-api-access-tmsj9\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.317377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.317361 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-metrics-certs\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.320869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.320843 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b1b214-72d8-4350-b6f4-bd7026b12a76-openshift-service-ca\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.326939 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.326913 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-metrics-certs\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.342627 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.342595 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmsj9\" (UniqueName: \"kubernetes.io/projected/87b1b214-72d8-4350-b6f4-bd7026b12a76-kube-api-access-tmsj9\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.343619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.343590 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-maas-api-tls\") pod \"maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.350657 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.350614 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:55.378227 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.378200 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:55.724794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.719204 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt"] Oct 31 01:47:55.762293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.760619 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:55.922385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:55.922346 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:56.073299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:56.073260 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:56.192372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:56.192341 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" event={"ID":"87b1b214-72d8-4350-b6f4-bd7026b12a76","Type":"ContainerStarted","Data":"f387ee59de5f061710672c0a926bdd1a915d092b7017d26c606a3c2783de13d5"} Oct 31 01:47:56.242086 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:56.242057 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:56.413585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:56.413507 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:56.584757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:56.584637 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:56.969838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:56.969800 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:57.200377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:57.200340 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" event={"ID":"87b1b214-72d8-4350-b6f4-bd7026b12a76","Type":"ContainerStarted","Data":"b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a"} Oct 31 01:47:57.200901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:57.200577 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:47:57.225053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:57.224954 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" podStartSLOduration=1.7898368260000002 podStartE2EDuration="2.224939532s" totalImagesPullingTime="435.102706ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:47:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:47:57.223109558 +0000 UTC m=+1677.603354793" watchObservedRunningTime="2026-10-31 01:47:57.224939532 +0000 UTC m=+1677.605184754" Oct 31 01:47:57.678735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:57.678351 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:58.567389 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:58.567334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:59.421575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:59.421542 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:47:59.949732 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:47:59.949684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:00.237507 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:00.237481 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:00.418433 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:00.418398 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:01.194073 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:01.194045 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:01.417742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:01.411689 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:01.557307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:01.557276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:01.996822 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:01.996777 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:01.997790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:01.997766 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp"] Oct 31 01:48:02.003643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.003625 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.009136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.009116 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9f0694\"" Oct 31 01:48:02.009342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.009318 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f0694-dockercfg-4qfhq\"" Oct 31 01:48:02.013781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.013759 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp"] Oct 31 01:48:02.041051 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.041012 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjmb\" (UniqueName: \"kubernetes.io/projected/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-kube-api-access-qsjmb\") pod \"payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.041239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.041151 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.157230 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.153588 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjmb\" (UniqueName: \"kubernetes.io/projected/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-kube-api-access-qsjmb\") pod \"payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.157230 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.153675 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.157230 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.154762 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.192771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.189473 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:02.224500 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.222675 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjmb\" (UniqueName: \"kubernetes.io/projected/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-kube-api-access-qsjmb\") pod \"payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.320621 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.320364 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:02.347549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.347406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:02.615857 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.615828 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp"] Oct 31 01:48:02.617407 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:02.617377 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7991bc_423a_4b94_a8ff_4ec0a80f8e9c.slice/crio-13c0bba9793f286b9a10aced7bcbbf1fa21cf61ebafd903165a0f7c812727ce7 WatchSource:0}: Error finding container 13c0bba9793f286b9a10aced7bcbbf1fa21cf61ebafd903165a0f7c812727ce7: Status 404 returned error can't find the container with id 13c0bba9793f286b9a10aced7bcbbf1fa21cf61ebafd903165a0f7c812727ce7 Oct 31 01:48:02.977691 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:02.975018 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:03.221871 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:03.221839 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:48:03.305774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:03.305703 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" event={"ID":"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c","Type":"ContainerStarted","Data":"13c0bba9793f286b9a10aced7bcbbf1fa21cf61ebafd903165a0f7c812727ce7"} Oct 31 01:48:03.904173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:03.904112 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:04.646595 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:04.644342 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:04.796054 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:48:04.796011 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:48:05.372750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:05.371445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:06.043778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.042148 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:06.338483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.338380 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" event={"ID":"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c","Type":"ContainerStarted","Data":"fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56"} Oct 31 01:48:06.338630 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.338594 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:06.357581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.357532 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" podStartSLOduration=2.728997195 podStartE2EDuration="5.357519618s" totalImagesPullingTime="2.628522423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:06.35419789 +0000 UTC m=+1686.734443165" watchObservedRunningTime="2026-10-31 01:48:06.357519618 +0000 UTC m=+1686.737764841" Oct 31 01:48:06.661648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.661570 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j"] Oct 31 01:48:06.667499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.667474 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.675224 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.675197 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-03004a-serving-cert\"" Oct 31 01:48:06.675823 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.675796 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j"] Oct 31 01:48:06.743182 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.743090 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:06.748772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.748735 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11422-7a414d"] Oct 31 01:48:06.753841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.753810 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:06.757452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.757424 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11422-7a414d"] Oct 31 01:48:06.758117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.757767 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzt9\" (UniqueName: \"kubernetes.io/projected/d4c2635d-a69a-462e-97eb-04a0ced1a872-kube-api-access-cxzt9\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.758117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.757992 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-metrics-certs\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.758117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.758077 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-maas-api-tls\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.758319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.758125 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c2635d-a69a-462e-97eb-04a0ced1a872-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.769172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.769146 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmw2f\"" Oct 31 01:48:06.859606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.859571 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzt9\" (UniqueName: \"kubernetes.io/projected/d4c2635d-a69a-462e-97eb-04a0ced1a872-kube-api-access-cxzt9\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.859793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.859635 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpzb\" (UniqueName: \"kubernetes.io/projected/8e3302cb-228a-4811-9eb4-9eb11c3e8c22-kube-api-access-wrpzb\") pod \"test-curl-11422-7a414d\" (UID: \"8e3302cb-228a-4811-9eb4-9eb11c3e8c22\") " pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:06.859846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.859798 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-metrics-certs\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.859897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.859850 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-maas-api-tls\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.859943 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.859921 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c2635d-a69a-462e-97eb-04a0ced1a872-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.860629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.860547 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c2635d-a69a-462e-97eb-04a0ced1a872-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.862341 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.862320 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-metrics-certs\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.862436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.862392 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-maas-api-tls\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.871631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.871606 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzt9\" (UniqueName: \"kubernetes.io/projected/d4c2635d-a69a-462e-97eb-04a0ced1a872-kube-api-access-cxzt9\") pod \"maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:06.920794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.919821 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:06.966366 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.966286 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpzb\" (UniqueName: \"kubernetes.io/projected/8e3302cb-228a-4811-9eb4-9eb11c3e8c22-kube-api-access-wrpzb\") pod \"test-curl-11422-7a414d\" (UID: \"8e3302cb-228a-4811-9eb4-9eb11c3e8c22\") " pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:06.978878 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:06.978848 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpzb\" (UniqueName: \"kubernetes.io/projected/8e3302cb-228a-4811-9eb4-9eb11c3e8c22-kube-api-access-wrpzb\") pod \"test-curl-11422-7a414d\" (UID: \"8e3302cb-228a-4811-9eb4-9eb11c3e8c22\") " pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:07.001128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.001089 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:07.070412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.070330 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:07.075186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.074939 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:07.285577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.285517 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j"] Oct 31 01:48:07.287265 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:07.287237 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c2635d_a69a_462e_97eb_04a0ced1a872.slice/crio-ed8827a7a484ce2023084c6bc5e19bfcde66ca57edbc2c02f05f74968e6cc814 WatchSource:0}: Error finding container ed8827a7a484ce2023084c6bc5e19bfcde66ca57edbc2c02f05f74968e6cc814: Status 404 returned error can't find the container with id ed8827a7a484ce2023084c6bc5e19bfcde66ca57edbc2c02f05f74968e6cc814 Oct 31 01:48:07.387316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.387243 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" event={"ID":"d4c2635d-a69a-462e-97eb-04a0ced1a872","Type":"ContainerStarted","Data":"ed8827a7a484ce2023084c6bc5e19bfcde66ca57edbc2c02f05f74968e6cc814"} Oct 31 01:48:07.412708 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.412675 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11422-7a414d"] Oct 31 01:48:07.413276 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:07.413234 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e3302cb_228a_4811_9eb4_9eb11c3e8c22.slice/crio-d20fc621c55c3ed9ad9ad981a54d2e99dcf8eb19812eb6de8b8e70a5f9e75228 WatchSource:0}: Error finding container d20fc621c55c3ed9ad9ad981a54d2e99dcf8eb19812eb6de8b8e70a5f9e75228: Status 404 returned error can't find the container with id d20fc621c55c3ed9ad9ad981a54d2e99dcf8eb19812eb6de8b8e70a5f9e75228 Oct 31 01:48:07.654043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.650123 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw"] Oct 31 01:48:07.657094 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.655533 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.658995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.658956 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9f0694-serving-cert\"" Oct 31 01:48:07.663257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.662769 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw"] Oct 31 01:48:07.757783 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.757751 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:07.790114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.783566 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.790114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.783742 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwlb\" (UniqueName: \"kubernetes.io/projected/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-kube-api-access-hlwlb\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.790114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.783810 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.790114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.783860 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.884906 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.884862 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.885343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.885320 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwlb\" (UniqueName: \"kubernetes.io/projected/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-kube-api-access-hlwlb\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.885642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.885525 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.886323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.886296 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.887340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.887313 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.889236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.889204 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-metrics-certs\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.890117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.890098 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.897525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.897500 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwlb\" (UniqueName: \"kubernetes.io/projected/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-kube-api-access-hlwlb\") pod \"maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:07.913389 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.913290 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:07.975101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:07.974868 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:08.304579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.299487 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw"] Oct 31 01:48:08.317914 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:08.312983 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc35581_a4a1_4ac2_9d93_f54ca34d378f.slice/crio-9b4d80f0dc89bdf2d3c6edf014c435ca801fb706d69a6ac54825237182d3341c WatchSource:0}: Error finding container 9b4d80f0dc89bdf2d3c6edf014c435ca801fb706d69a6ac54825237182d3341c: Status 404 returned error can't find the container with id 9b4d80f0dc89bdf2d3c6edf014c435ca801fb706d69a6ac54825237182d3341c Oct 31 01:48:08.412750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.412079 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" event={"ID":"d4c2635d-a69a-462e-97eb-04a0ced1a872","Type":"ContainerStarted","Data":"7ac0d7e0b2003cff7011f57d3fca7c25d7fe5c593e06ec5b7fb582464035f49b"} Oct 31 01:48:08.412750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.412731 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:08.415586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.415514 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" event={"ID":"ecc35581-a4a1-4ac2-9d93-f54ca34d378f","Type":"ContainerStarted","Data":"9b4d80f0dc89bdf2d3c6edf014c435ca801fb706d69a6ac54825237182d3341c"} Oct 31 01:48:08.418586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.418560 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11422-7a414d" event={"ID":"8e3302cb-228a-4811-9eb4-9eb11c3e8c22","Type":"ContainerStarted","Data":"e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a"} Oct 31 01:48:08.418766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.418592 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11422-7a414d" event={"ID":"8e3302cb-228a-4811-9eb4-9eb11c3e8c22","Type":"ContainerStarted","Data":"d20fc621c55c3ed9ad9ad981a54d2e99dcf8eb19812eb6de8b8e70a5f9e75228"} Oct 31 01:48:08.435244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.435189 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" podStartSLOduration=1.8755274530000001 podStartE2EDuration="2.4351662s" totalImagesPullingTime="559.638747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:08.43352119 +0000 UTC m=+1688.813766423" watchObservedRunningTime="2026-10-31 01:48:08.4351662 +0000 UTC m=+1688.815411426" Oct 31 01:48:08.452649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.452600 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11422-7a414d" podStartSLOduration=2.452587743 podStartE2EDuration="2.452587743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:08.450471789 +0000 UTC m=+1688.830717032" watchObservedRunningTime="2026-10-31 01:48:08.452587743 +0000 UTC m=+1688.832832966" Oct 31 01:48:08.576942 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:08.576849 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:09.146854 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:09.146826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:09.286277 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:09.286245 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11422-7a414d"] Oct 31 01:48:09.294379 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:09.294345 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11422-7a414d"] Oct 31 01:48:09.380171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:09.380135 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:09.429263 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:09.429222 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" event={"ID":"ecc35581-a4a1-4ac2-9d93-f54ca34d378f","Type":"ContainerStarted","Data":"13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73"} Oct 31 01:48:09.431141 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:09.430514 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:09.454056 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:09.454000 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" podStartSLOduration=1.878483724 podStartE2EDuration="2.45398561s" totalImagesPullingTime="575.501886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:09.450819286 +0000 UTC m=+1689.831064526" watchObservedRunningTime="2026-10-31 01:48:09.45398561 +0000 UTC m=+1689.834230832" Oct 31 01:48:10.360748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:10.358855 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:10.437355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:10.437309 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11422-7a414d" podUID="8e3302cb-228a-4811-9eb4-9eb11c3e8c22" containerName="test-curl-11422-7a414d" containerID="cri-o://e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a" gracePeriod=2 Oct 31 01:48:11.201284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:11.201241 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:11.753022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:11.752978 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:12.341553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:12.341526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:12.622860 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:12.622836 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:12.701055 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:12.701026 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e3302cb-228a-4811-9eb4-9eb11c3e8c22-kube-api-access-wrpzb\" (UniqueName: \"kubernetes.io/projected/8e3302cb-228a-4811-9eb4-9eb11c3e8c22-kube-api-access-wrpzb\") pod \"8e3302cb-228a-4811-9eb4-9eb11c3e8c22\" (UID: \"8e3302cb-228a-4811-9eb4-9eb11c3e8c22\") " Oct 31 01:48:12.703341 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:12.703311 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e3302cb-228a-4811-9eb4-9eb11c3e8c22-kube-api-access-wrpzb" pod "8e3302cb-228a-4811-9eb4-9eb11c3e8c22" (UID: "8e3302cb-228a-4811-9eb4-9eb11c3e8c22"). InnerVolumeSpecName "kube-api-access-wrpzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:12.802800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:12.802769 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrpzb\" (UniqueName: \"kubernetes.io/projected/8e3302cb-228a-4811-9eb4-9eb11c3e8c22-kube-api-access-wrpzb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:13.029737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.021174 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:13.465272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.465193 5090 generic.go:356] "Generic (PLEG): container finished" podID="8e3302cb-228a-4811-9eb4-9eb11c3e8c22" containerID="e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a" exitCode=137 Oct 31 01:48:13.465272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.465236 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:13.465272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.465264 5090 scope.go:122] "RemoveContainer" containerID="e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a" Oct 31 01:48:13.474921 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:48:13.474893 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11422-7a414d\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="8e3302cb-228a-4811-9eb4-9eb11c3e8c22" pod="opendatahub/test-curl-11422-7a414d" Oct 31 01:48:13.475117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.475103 5090 scope.go:122] "RemoveContainer" containerID="e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a" Oct 31 01:48:13.475363 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:48:13.475346 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a\": container with ID starting with e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a not found: ID does not exist" containerID="e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a" Oct 31 01:48:13.475424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.475372 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a"} err="failed to get container status \"e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a\": rpc error: code = NotFound desc = could not find container \"e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a\": container with ID starting with e896e88e57648f00fe98ae943bf79e5c021e1a6281d80b0ea0a1ac4680d4ea0a not found: ID does not exist" Oct 31 01:48:13.768024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.767509 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e3302cb-228a-4811-9eb4-9eb11c3e8c22" path="/var/lib/kubelet/pods/8e3302cb-228a-4811-9eb4-9eb11c3e8c22/volumes" Oct 31 01:48:13.895538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:13.895502 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:14.439755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:14.439702 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:48:14.986409 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:14.986366 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:15.457286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:15.457258 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:48:16.183727 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:16.183668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:17.023814 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:17.023769 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:17.392517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:17.392426 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:18.311740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:18.311686 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:18.966254 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:18.966218 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:19.652384 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:19.652232 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:19.799009 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:19.798974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:19.950519 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:19.950437 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:20.283399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:20.283371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:20.419394 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:20.419362 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp"] Oct 31 01:48:20.427293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:20.427260 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:20.587274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:20.587167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:20.730114 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:20.730065 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:20.922667 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:20.910078 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:21.479063 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:21.474875 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:21.759645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:21.759614 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:21.902248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:21.902219 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:22.136429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:22.132448 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:22.671770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:22.671734 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj"] Oct 31 01:48:22.672250 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:22.672037 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" podUID="8430f292-950d-4f71-b3b0-490829b0f473" containerName="maas-api" containerID="cri-o://2bbb50ef467768b9e75b71ee74dfe5311eba31510c006fe59400527e0b33edb5" gracePeriod=30 Oct 31 01:48:22.761270 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:22.761232 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:22.915241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:22.915211 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:23.063299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.063218 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:23.072266 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.071827 5090 generic.go:356] "Generic (PLEG): container finished" podID="8430f292-950d-4f71-b3b0-490829b0f473" containerID="2bbb50ef467768b9e75b71ee74dfe5311eba31510c006fe59400527e0b33edb5" exitCode=0 Oct 31 01:48:23.072266 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.071983 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" event={"ID":"8430f292-950d-4f71-b3b0-490829b0f473","Type":"ContainerDied","Data":"2bbb50ef467768b9e75b71ee74dfe5311eba31510c006fe59400527e0b33edb5"} Oct 31 01:48:23.110545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.110214 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:48:23.169239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.169213 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-metrics-certs\") pod \"8430f292-950d-4f71-b3b0-490829b0f473\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " Oct 31 01:48:23.169575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.169561 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8430f292-950d-4f71-b3b0-490829b0f473-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8430f292-950d-4f71-b3b0-490829b0f473-openshift-service-ca\") pod \"8430f292-950d-4f71-b3b0-490829b0f473\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " Oct 31 01:48:23.169686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.169675 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls\") pod \"8430f292-950d-4f71-b3b0-490829b0f473\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " Oct 31 01:48:23.169816 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.169797 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8430f292-950d-4f71-b3b0-490829b0f473-kube-api-access-jm4xq\" (UniqueName: \"kubernetes.io/projected/8430f292-950d-4f71-b3b0-490829b0f473-kube-api-access-jm4xq\") pod \"8430f292-950d-4f71-b3b0-490829b0f473\" (UID: \"8430f292-950d-4f71-b3b0-490829b0f473\") " Oct 31 01:48:23.170836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.170384 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8430f292-950d-4f71-b3b0-490829b0f473-openshift-service-ca" pod "8430f292-950d-4f71-b3b0-490829b0f473" (UID: "8430f292-950d-4f71-b3b0-490829b0f473"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:48:23.171149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.171124 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8430f292-950d-4f71-b3b0-490829b0f473-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:23.172995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.172953 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls" pod "8430f292-950d-4f71-b3b0-490829b0f473" (UID: "8430f292-950d-4f71-b3b0-490829b0f473"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:23.173412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.173359 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8430f292-950d-4f71-b3b0-490829b0f473-kube-api-access-jm4xq" pod "8430f292-950d-4f71-b3b0-490829b0f473" (UID: "8430f292-950d-4f71-b3b0-490829b0f473"). InnerVolumeSpecName "kube-api-access-jm4xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:23.176021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.175999 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-metrics-certs" pod "8430f292-950d-4f71-b3b0-490829b0f473" (UID: "8430f292-950d-4f71-b3b0-490829b0f473"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:23.258616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.258589 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:23.274741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.273908 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:23.274741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.273940 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8430f292-950d-4f71-b3b0-490829b0f473-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:23.274741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.273972 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4xq\" (UniqueName: \"kubernetes.io/projected/8430f292-950d-4f71-b3b0-490829b0f473-kube-api-access-jm4xq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:23.496897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:23.496872 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:24.079946 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.079858 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" Oct 31 01:48:24.079946 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.079901 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj" event={"ID":"8430f292-950d-4f71-b3b0-490829b0f473","Type":"ContainerDied","Data":"8628bc5c36fec6c7e2f712df52a6853123e3a365fa844943a58066c9474005c5"} Oct 31 01:48:24.080536 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.079973 5090 scope.go:122] "RemoveContainer" containerID="2bbb50ef467768b9e75b71ee74dfe5311eba31510c006fe59400527e0b33edb5" Oct 31 01:48:24.099501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.099462 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj"] Oct 31 01:48:24.102981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.102955 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cb5756cb-6d54df8fd9-n22nj"] Oct 31 01:48:24.249078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.248533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:24.699345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.699299 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:24.743517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.743490 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4"] Oct 31 01:48:24.744173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.744157 5090 cpu_manager.go:414] "containerMap: removing container" podUID="8430f292-950d-4f71-b3b0-490829b0f473" containerName="maas-api" Oct 31 01:48:24.744245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.744174 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8430f292-950d-4f71-b3b0-490829b0f473" containerName="maas-api" Oct 31 01:48:24.744245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.744184 5090 cpu_manager.go:414] "containerMap: removing container" podUID="8e3302cb-228a-4811-9eb4-9eb11c3e8c22" containerName="test-curl-11422-7a414d" Oct 31 01:48:24.744245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.744190 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e3302cb-228a-4811-9eb4-9eb11c3e8c22" containerName="test-curl-11422-7a414d" Oct 31 01:48:24.744338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.744280 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="8430f292-950d-4f71-b3b0-490829b0f473" containerName="maas-api" Oct 31 01:48:24.744338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.744288 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e3302cb-228a-4811-9eb4-9eb11c3e8c22" containerName="test-curl-11422-7a414d" Oct 31 01:48:24.749002 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.748984 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:24.759000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.758977 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-03004a\"" Oct 31 01:48:24.759134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.759108 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-03004a-dockercfg-r7tnb\"" Oct 31 01:48:24.761661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.761640 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4"] Oct 31 01:48:24.793606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.793573 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99"] Oct 31 01:48:24.798575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.798554 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:24.810325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.810201 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99"] Oct 31 01:48:24.882273 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.882235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:24.929742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.924060 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvcx\" (UniqueName: \"kubernetes.io/projected/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-kube-api-access-rtvcx\") pod \"payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:24.929742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.924215 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9qn\" (UniqueName: \"kubernetes.io/projected/9bca4fe6-189f-4336-a71a-2e4ce715ab32-kube-api-access-cq9qn\") pod \"payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:24.929742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.924262 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:24.929742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:24.924352 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bca4fe6-189f-4336-a71a-2e4ce715ab32-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:25.025992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.025955 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bca4fe6-189f-4336-a71a-2e4ce715ab32-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:25.026221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.026086 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvcx\" (UniqueName: \"kubernetes.io/projected/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-kube-api-access-rtvcx\") pod \"payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:25.026221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.026186 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9qn\" (UniqueName: \"kubernetes.io/projected/9bca4fe6-189f-4336-a71a-2e4ce715ab32-kube-api-access-cq9qn\") pod \"payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:25.026221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.026221 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:25.026849 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.026821 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bca4fe6-189f-4336-a71a-2e4ce715ab32-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:25.027269 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.027241 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:25.039237 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.039198 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvcx\" (UniqueName: \"kubernetes.io/projected/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-kube-api-access-rtvcx\") pod \"payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:25.039395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.039367 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9qn\" (UniqueName: \"kubernetes.io/projected/9bca4fe6-189f-4336-a71a-2e4ce715ab32-kube-api-access-cq9qn\") pod \"payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:25.065405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.065103 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:25.116572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.116539 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:25.320842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.315631 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:25.359911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.357845 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4"] Oct 31 01:48:25.362101 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:25.362071 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e15e12_049d_4c0d_a3e7_bd56dda3e40c.slice/crio-65b453182163888bcab054763f1ce6eb4aeaed2f56be84c3908da1f47574b5d2 WatchSource:0}: Error finding container 65b453182163888bcab054763f1ce6eb4aeaed2f56be84c3908da1f47574b5d2: Status 404 returned error can't find the container with id 65b453182163888bcab054763f1ce6eb4aeaed2f56be84c3908da1f47574b5d2 Oct 31 01:48:25.420182 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.420126 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" podUID="9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" containerName="payload-processing" containerID="cri-o://fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56" gracePeriod=25 Oct 31 01:48:25.449198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.449172 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99"] Oct 31 01:48:25.466538 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:25.466512 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bca4fe6_189f_4336_a71a_2e4ce715ab32.slice/crio-e30c4bbbd00b95bf2c3c02bc17c4f43eb04c758b4294c50303607d9ddddf628e WatchSource:0}: Error finding container e30c4bbbd00b95bf2c3c02bc17c4f43eb04c758b4294c50303607d9ddddf628e: Status 404 returned error can't find the container with id e30c4bbbd00b95bf2c3c02bc17c4f43eb04c758b4294c50303607d9ddddf628e Oct 31 01:48:25.680944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.680913 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:25.751597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.751561 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8430f292-950d-4f71-b3b0-490829b0f473" path="/var/lib/kubelet/pods/8430f292-950d-4f71-b3b0-490829b0f473/volumes" Oct 31 01:48:25.833942 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.833889 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:25.839629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.839603 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-plugins-config-volume\") pod \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " Oct 31 01:48:25.839770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.839754 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-kube-api-access-qsjmb\" (UniqueName: \"kubernetes.io/projected/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-kube-api-access-qsjmb\") pod \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\" (UID: \"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c\") " Oct 31 01:48:25.840069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.840047 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-plugins-config-volume" pod "9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" (UID: "9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:48:25.840510 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.840446 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:25.841964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.841944 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-kube-api-access-qsjmb" pod "9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" (UID: "9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c"). InnerVolumeSpecName "kube-api-access-qsjmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:25.885811 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.885729 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm"] Oct 31 01:48:25.886813 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.886791 5090 cpu_manager.go:414] "containerMap: removing container" podUID="9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" containerName="payload-processing" Oct 31 01:48:25.886813 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.886814 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" containerName="payload-processing" Oct 31 01:48:25.887031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.887020 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" containerName="payload-processing" Oct 31 01:48:25.890970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.890952 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:25.908264 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.908240 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm"] Oct 31 01:48:25.942227 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.942197 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjmb\" (UniqueName: \"kubernetes.io/projected/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c-kube-api-access-qsjmb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:25.967173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.967135 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9f0694-dockercfg-nlb72\"" Oct 31 01:48:25.967499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:25.967481 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:26.062036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.048364 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7823fadd-da41-479a-87d3-83631ff40970-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:26.104065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.104008 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv5jx\" (UniqueName: \"kubernetes.io/projected/7823fadd-da41-479a-87d3-83631ff40970-kube-api-access-kv5jx\") pod \"payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:26.137124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.136981 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" event={"ID":"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c","Type":"ContainerStarted","Data":"65b453182163888bcab054763f1ce6eb4aeaed2f56be84c3908da1f47574b5d2"} Oct 31 01:48:26.153830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.151898 5090 generic.go:356] "Generic (PLEG): container finished" podID="9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" containerID="fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56" exitCode=0 Oct 31 01:48:26.153830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.152064 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" event={"ID":"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c","Type":"ContainerDied","Data":"fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56"} Oct 31 01:48:26.153830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.152119 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" event={"ID":"9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c","Type":"ContainerDied","Data":"13c0bba9793f286b9a10aced7bcbbf1fa21cf61ebafd903165a0f7c812727ce7"} Oct 31 01:48:26.153830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.152156 5090 scope.go:122] "RemoveContainer" containerID="fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56" Oct 31 01:48:26.153830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.152314 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp" Oct 31 01:48:26.176677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.176553 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:26.184561 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.184447 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" event={"ID":"9bca4fe6-189f-4336-a71a-2e4ce715ab32","Type":"ContainerStarted","Data":"e30c4bbbd00b95bf2c3c02bc17c4f43eb04c758b4294c50303607d9ddddf628e"} Oct 31 01:48:26.215616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.214674 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv5jx\" (UniqueName: \"kubernetes.io/projected/7823fadd-da41-479a-87d3-83631ff40970-kube-api-access-kv5jx\") pod \"payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:26.216976 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.216873 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7823fadd-da41-479a-87d3-83631ff40970-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:26.217944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.217918 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7823fadd-da41-479a-87d3-83631ff40970-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:26.219053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.218948 5090 scope.go:122] "RemoveContainer" containerID="fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56" Oct 31 01:48:26.219492 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:48:26.219419 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56\": container with ID starting with fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56 not found: ID does not exist" containerID="fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56" Oct 31 01:48:26.219492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.219454 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56"} err="failed to get container status \"fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56\": rpc error: code = NotFound desc = could not find container \"fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56\": container with ID starting with fa9b073f2b85fc056b03dd2233e7f6a6e9159fa8548da1e21121b0f0a4df0b56 not found: ID does not exist" Oct 31 01:48:26.220364 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.220338 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp"] Oct 31 01:48:26.230613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.230565 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-5556cd996f-9tsrp"] Oct 31 01:48:26.231235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.231211 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv5jx\" (UniqueName: \"kubernetes.io/projected/7823fadd-da41-479a-87d3-83631ff40970-kube-api-access-kv5jx\") pod \"payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:26.367739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.366936 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:26.520771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.520745 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:26.522754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.522671 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:26.869539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:26.869508 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm"] Oct 31 01:48:26.874782 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:26.874746 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7823fadd_da41_479a_87d3_83631ff40970.slice/crio-62c04025b12c6ce6f68e28c31326b10a1dee045502937a2a434a84798aaf6fbe WatchSource:0}: Error finding container 62c04025b12c6ce6f68e28c31326b10a1dee045502937a2a434a84798aaf6fbe: Status 404 returned error can't find the container with id 62c04025b12c6ce6f68e28c31326b10a1dee045502937a2a434a84798aaf6fbe Oct 31 01:48:27.191294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:27.190072 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:27.203380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:27.203346 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" event={"ID":"7823fadd-da41-479a-87d3-83631ff40970","Type":"ContainerStarted","Data":"62c04025b12c6ce6f68e28c31326b10a1dee045502937a2a434a84798aaf6fbe"} Oct 31 01:48:27.778396 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:27.778354 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c" path="/var/lib/kubelet/pods/9f7991bc-423a-4b94-a8ff-4ec0a80f8e9c/volumes" Oct 31 01:48:27.930454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:27.930394 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:28.370402 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:28.370290 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss"] Oct 31 01:48:28.372181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:28.370516 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" podUID="899abbef-2312-483f-b16c-8cc2728a31e2" containerName="maas-api" containerID="cri-o://328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b" gracePeriod=30 Oct 31 01:48:28.477090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:28.476980 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:28.633755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:28.629692 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:28.799457 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:28.799386 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:29.050098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.050005 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:29.056289 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.055798 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:48:29.123661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.122536 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-maas-api-tls\") pod \"899abbef-2312-483f-b16c-8cc2728a31e2\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " Oct 31 01:48:29.123661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.122616 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-metrics-certs\") pod \"899abbef-2312-483f-b16c-8cc2728a31e2\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " Oct 31 01:48:29.123661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.122831 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/899abbef-2312-483f-b16c-8cc2728a31e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/899abbef-2312-483f-b16c-8cc2728a31e2-openshift-service-ca\") pod \"899abbef-2312-483f-b16c-8cc2728a31e2\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " Oct 31 01:48:29.123661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.122906 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/899abbef-2312-483f-b16c-8cc2728a31e2-kube-api-access-2gf97\" (UniqueName: \"kubernetes.io/projected/899abbef-2312-483f-b16c-8cc2728a31e2-kube-api-access-2gf97\") pod \"899abbef-2312-483f-b16c-8cc2728a31e2\" (UID: \"899abbef-2312-483f-b16c-8cc2728a31e2\") " Oct 31 01:48:29.125165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.125118 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899abbef-2312-483f-b16c-8cc2728a31e2-openshift-service-ca" pod "899abbef-2312-483f-b16c-8cc2728a31e2" (UID: "899abbef-2312-483f-b16c-8cc2728a31e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:48:29.135944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.135919 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-metrics-certs" pod "899abbef-2312-483f-b16c-8cc2728a31e2" (UID: "899abbef-2312-483f-b16c-8cc2728a31e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:29.140889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.139519 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899abbef-2312-483f-b16c-8cc2728a31e2-kube-api-access-2gf97" pod "899abbef-2312-483f-b16c-8cc2728a31e2" (UID: "899abbef-2312-483f-b16c-8cc2728a31e2"). InnerVolumeSpecName "kube-api-access-2gf97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:29.142594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.142548 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-maas-api-tls" pod "899abbef-2312-483f-b16c-8cc2728a31e2" (UID: "899abbef-2312-483f-b16c-8cc2728a31e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:29.226785 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.226704 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" event={"ID":"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c","Type":"ContainerStarted","Data":"f696cf3ea76be5dbd80e3ff4213781354b94cabab15f4a33fa0613c8aca1039f"} Oct 31 01:48:29.227816 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.227792 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:29.229369 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.229291 5090 generic.go:356] "Generic (PLEG): container finished" podID="899abbef-2312-483f-b16c-8cc2728a31e2" containerID="328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b" exitCode=0 Oct 31 01:48:29.229489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.229411 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" event={"ID":"899abbef-2312-483f-b16c-8cc2728a31e2","Type":"ContainerDied","Data":"328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b"} Oct 31 01:48:29.229489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.229448 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" event={"ID":"899abbef-2312-483f-b16c-8cc2728a31e2","Type":"ContainerDied","Data":"425884615b9a8072a5a3a3bec967edb1b0080888c9eb8562a2ef2e45691441ee"} Oct 31 01:48:29.229489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.229478 5090 scope.go:122] "RemoveContainer" containerID="328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b" Oct 31 01:48:29.230204 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.229623 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss" Oct 31 01:48:29.235143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.234811 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/899abbef-2312-483f-b16c-8cc2728a31e2-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:29.235143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.234838 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gf97\" (UniqueName: \"kubernetes.io/projected/899abbef-2312-483f-b16c-8cc2728a31e2-kube-api-access-2gf97\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:29.235143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.234855 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:29.235143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.234869 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/899abbef-2312-483f-b16c-8cc2728a31e2-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:29.237062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.237036 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" event={"ID":"9bca4fe6-189f-4336-a71a-2e4ce715ab32","Type":"ContainerStarted","Data":"0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a"} Oct 31 01:48:29.238273 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.237829 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:29.248107 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.248062 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" podStartSLOduration=2.4156393290000002 podStartE2EDuration="5.24804672s" totalImagesPullingTime="2.832407391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:29.246552368 +0000 UTC m=+1709.626797613" watchObservedRunningTime="2026-10-31 01:48:29.24804672 +0000 UTC m=+1709.628291948" Oct 31 01:48:29.266734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.266681 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" podStartSLOduration=2.531629656 podStartE2EDuration="5.266669051s" totalImagesPullingTime="2.735039395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:29.265950925 +0000 UTC m=+1709.646196169" watchObservedRunningTime="2026-10-31 01:48:29.266669051 +0000 UTC m=+1709.646914273" Oct 31 01:48:29.268221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.268173 5090 scope.go:122] "RemoveContainer" containerID="328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b" Oct 31 01:48:29.269552 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:48:29.268878 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b\": container with ID starting with 328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b not found: ID does not exist" containerID="328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b" Oct 31 01:48:29.269552 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.268906 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b"} err="failed to get container status \"328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b\": rpc error: code = NotFound desc = could not find container \"328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b\": container with ID starting with 328f7696819c7e3f8d8f46d06e4cedd4fca1d635f3ab8b50f72c782a0ff1a78b not found: ID does not exist" Oct 31 01:48:29.288878 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.288813 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss"] Oct 31 01:48:29.294348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.294314 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-50ac12-cb4568b79-l7bss"] Oct 31 01:48:29.357525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.357440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:29.749001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.748965 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="899abbef-2312-483f-b16c-8cc2728a31e2" path="/var/lib/kubelet/pods/899abbef-2312-483f-b16c-8cc2728a31e2/volumes" Oct 31 01:48:29.872446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:29.872414 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:30.247799 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:30.247766 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" event={"ID":"7823fadd-da41-479a-87d3-83631ff40970","Type":"ContainerStarted","Data":"167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70"} Oct 31 01:48:30.248636 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:30.248616 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:30.267070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:30.267021 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" podStartSLOduration=3.029280062 podStartE2EDuration="5.267004837s" totalImagesPullingTime="2.237724775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:30.265361913 +0000 UTC m=+1710.645607154" watchObservedRunningTime="2026-10-31 01:48:30.267004837 +0000 UTC m=+1710.647250061" Oct 31 01:48:30.565815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:30.564965 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:31.234690 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:31.234647 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:32.137742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:32.137575 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:32.658865 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:32.658836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:33.008691 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:33.008661 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:33.153487 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:33.153452 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:33.938860 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:33.938824 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:34.652373 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:34.652336 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:34.789750 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:48:34.789679 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:48:35.514674 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:35.514638 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:35.680986 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:35.680956 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:35.830803 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:35.830191 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:36.118801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:36.118642 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:36.449763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:36.449660 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:37.322049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:37.322001 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:38.068791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:38.068759 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:38.741756 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:38.739482 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:39.527523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:39.527428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:40.537141 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:40.537101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:41.257951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:41.257924 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:48:41.258155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:41.258000 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:48:41.335467 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:41.335432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:42.010482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:42.010392 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:42.159416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:42.159384 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:42.264966 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:42.264882 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:48:42.509525 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:42.509492 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:42.647778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:42.647671 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:42.826320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:42.825930 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:43.423360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:43.423330 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:44.077193 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:44.077162 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:44.884929 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:44.884882 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:45.625656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:45.625622 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:46.374849 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:46.374811 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:46.629488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:46.629420 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:46.786921 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:46.786874 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:46.943633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:46.943563 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:47.079944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:47.079912 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:47.500429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:47.500396 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:47.644581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:47.644547 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:47.909959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:47.909876 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:48.096424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:48.096393 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:48.255354 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:48.255246 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:48.663303 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:48.662351 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:49.448331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:49.448285 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:50.182746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:50.182698 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:50.350332 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:50.350301 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:50.826107 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:50.826078 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:51.612643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:51.612607 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:52.116514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.116483 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:52.244076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.244044 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57"] Oct 31 01:48:52.244852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.244832 5090 cpu_manager.go:414] "containerMap: removing container" podUID="899abbef-2312-483f-b16c-8cc2728a31e2" containerName="maas-api" Oct 31 01:48:52.244852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.244853 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="899abbef-2312-483f-b16c-8cc2728a31e2" containerName="maas-api" Oct 31 01:48:52.245053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.245034 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="899abbef-2312-483f-b16c-8cc2728a31e2" containerName="maas-api" Oct 31 01:48:52.250291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.250267 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.266478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.266449 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-40a891-serving-cert\"" Oct 31 01:48:52.268893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.268730 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57"] Oct 31 01:48:52.297764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.286521 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:52.344591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.344560 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-metrics-certs\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.344787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.344606 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxfdq\" (UniqueName: \"kubernetes.io/projected/4700fe72-e630-46ff-b8fd-2adfa78b9600-kube-api-access-wxfdq\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.344787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.344700 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4700fe72-e630-46ff-b8fd-2adfa78b9600-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.344905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.344795 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-maas-api-tls\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.438554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.438443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:52.446835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.446804 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-metrics-certs\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.446996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.446861 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxfdq\" (UniqueName: \"kubernetes.io/projected/4700fe72-e630-46ff-b8fd-2adfa78b9600-kube-api-access-wxfdq\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.446996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.446905 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4700fe72-e630-46ff-b8fd-2adfa78b9600-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.446996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.446937 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-maas-api-tls\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.447734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.447685 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4700fe72-e630-46ff-b8fd-2adfa78b9600-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.449801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.449777 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-maas-api-tls\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.449928 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.449898 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-metrics-certs\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.460124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.460100 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxfdq\" (UniqueName: \"kubernetes.io/projected/4700fe72-e630-46ff-b8fd-2adfa78b9600-kube-api-access-wxfdq\") pod \"maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.578460 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.578428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:52.598262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.598234 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:52.889530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:52.889493 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57"] Oct 31 01:48:52.903767 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:52.897409 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4700fe72_e630_46ff_b8fd_2adfa78b9600.slice/crio-c5e1ab1d139ceb9b8d17ac04fb33198c583e2607b0a3d8e736fef7b2050f8817 WatchSource:0}: Error finding container c5e1ab1d139ceb9b8d17ac04fb33198c583e2607b0a3d8e736fef7b2050f8817: Status 404 returned error can't find the container with id c5e1ab1d139ceb9b8d17ac04fb33198c583e2607b0a3d8e736fef7b2050f8817 Oct 31 01:48:53.349562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:53.347259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:53.444922 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:53.444884 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" event={"ID":"4700fe72-e630-46ff-b8fd-2adfa78b9600","Type":"ContainerStarted","Data":"c5e1ab1d139ceb9b8d17ac04fb33198c583e2607b0a3d8e736fef7b2050f8817"} Oct 31 01:48:53.561778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:53.561751 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:53.793528 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:53.793043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:54.068483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:54.065238 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:54.303477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:54.303446 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:54.452708 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:54.452633 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" event={"ID":"4700fe72-e630-46ff-b8fd-2adfa78b9600","Type":"ContainerStarted","Data":"91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555"} Oct 31 01:48:54.452950 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:54.452932 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:48:54.489921 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:54.489877 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" podStartSLOduration=1.99954734 podStartE2EDuration="2.489863935s" totalImagesPullingTime="490.316595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:48:54.486557508 +0000 UTC m=+1734.866802751" watchObservedRunningTime="2026-10-31 01:48:54.489863935 +0000 UTC m=+1734.870109158" Oct 31 01:48:55.053812 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:55.053777 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:55.839897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:55.839866 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:56.542745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:56.542694 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:57.144739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:57.138954 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:57.625351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:57.625320 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt"] Oct 31 01:48:57.625620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:57.625593 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" podUID="87b1b214-72d8-4350-b6f4-bd7026b12a76" containerName="maas-api" containerID="cri-o://b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a" gracePeriod=30 Oct 31 01:48:57.697455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:57.697419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:57.851217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:57.851184 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:58.023846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.023680 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:58.102174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.101921 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:48:58.171586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.171541 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b1b214-72d8-4350-b6f4-bd7026b12a76-kube-api-access-tmsj9\" (UniqueName: \"kubernetes.io/projected/87b1b214-72d8-4350-b6f4-bd7026b12a76-kube-api-access-tmsj9\") pod \"87b1b214-72d8-4350-b6f4-bd7026b12a76\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " Oct 31 01:48:58.172299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.172260 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-metrics-certs\") pod \"87b1b214-72d8-4350-b6f4-bd7026b12a76\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " Oct 31 01:48:58.172517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.172376 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-maas-api-tls\") pod \"87b1b214-72d8-4350-b6f4-bd7026b12a76\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " Oct 31 01:48:58.172517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.172479 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b1b214-72d8-4350-b6f4-bd7026b12a76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b1b214-72d8-4350-b6f4-bd7026b12a76-openshift-service-ca\") pod \"87b1b214-72d8-4350-b6f4-bd7026b12a76\" (UID: \"87b1b214-72d8-4350-b6f4-bd7026b12a76\") " Oct 31 01:48:58.173147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.173100 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b1b214-72d8-4350-b6f4-bd7026b12a76-openshift-service-ca" pod "87b1b214-72d8-4350-b6f4-bd7026b12a76" (UID: "87b1b214-72d8-4350-b6f4-bd7026b12a76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:48:58.173625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.173601 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b1b214-72d8-4350-b6f4-bd7026b12a76-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:58.174498 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.174475 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:58.176565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.176539 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-maas-api-tls" pod "87b1b214-72d8-4350-b6f4-bd7026b12a76" (UID: "87b1b214-72d8-4350-b6f4-bd7026b12a76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:58.187240 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.187171 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b1b214-72d8-4350-b6f4-bd7026b12a76-kube-api-access-tmsj9" pod "87b1b214-72d8-4350-b6f4-bd7026b12a76" (UID: "87b1b214-72d8-4350-b6f4-bd7026b12a76"). InnerVolumeSpecName "kube-api-access-tmsj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:48:58.188603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.188578 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-metrics-certs" pod "87b1b214-72d8-4350-b6f4-bd7026b12a76" (UID: "87b1b214-72d8-4350-b6f4-bd7026b12a76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:48:58.283473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.279180 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmsj9\" (UniqueName: \"kubernetes.io/projected/87b1b214-72d8-4350-b6f4-bd7026b12a76-kube-api-access-tmsj9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:58.283473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.279221 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:58.283473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.279249 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b1b214-72d8-4350-b6f4-bd7026b12a76-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:48:58.427964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.427933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:58.481945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.481915 5090 generic.go:356] "Generic (PLEG): container finished" podID="87b1b214-72d8-4350-b6f4-bd7026b12a76" containerID="b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a" exitCode=0 Oct 31 01:48:58.482095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.481955 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" Oct 31 01:48:58.482095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.481993 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" event={"ID":"87b1b214-72d8-4350-b6f4-bd7026b12a76","Type":"ContainerDied","Data":"b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a"} Oct 31 01:48:58.482095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.482052 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt" event={"ID":"87b1b214-72d8-4350-b6f4-bd7026b12a76","Type":"ContainerDied","Data":"f387ee59de5f061710672c0a926bdd1a915d092b7017d26c606a3c2783de13d5"} Oct 31 01:48:58.482095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.482075 5090 scope.go:122] "RemoveContainer" containerID="b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a" Oct 31 01:48:58.492410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.492394 5090 scope.go:122] "RemoveContainer" containerID="b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a" Oct 31 01:48:58.492641 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:48:58.492621 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a\": container with ID starting with b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a not found: ID does not exist" containerID="b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a" Oct 31 01:48:58.492704 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.492652 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a"} err="failed to get container status \"b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a\": rpc error: code = NotFound desc = could not find container \"b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a\": container with ID starting with b1105e7e00286213a91e3cecaab3aafefba913ba17ceb5a0c5e38e9c319e063a not found: ID does not exist" Oct 31 01:48:58.506827 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.506795 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt"] Oct 31 01:48:58.512973 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.512953 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ea6cc6-84c5f98b9d-8zxdt"] Oct 31 01:48:58.885983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.885949 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9"] Oct 31 01:48:58.887030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.887006 5090 cpu_manager.go:414] "containerMap: removing container" podUID="87b1b214-72d8-4350-b6f4-bd7026b12a76" containerName="maas-api" Oct 31 01:48:58.887120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.887034 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b1b214-72d8-4350-b6f4-bd7026b12a76" containerName="maas-api" Oct 31 01:48:58.887224 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.887209 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b1b214-72d8-4350-b6f4-bd7026b12a76" containerName="maas-api" Oct 31 01:48:58.893474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.893449 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:58.902391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.902363 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9"] Oct 31 01:48:58.937160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.937136 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cbbaec-serving-cert\"" Oct 31 01:48:58.965802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.965776 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:58.997909 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.997687 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:58.997909 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:58.997795 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9jm\" (UniqueName: \"kubernetes.io/projected/a34254f0-0121-44f1-8ef3-27dda775a50d-kube-api-access-2s9jm\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.001680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.000806 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.001680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.000864 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34254f0-0121-44f1-8ef3-27dda775a50d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.103494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.103390 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.103675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.103524 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9jm\" (UniqueName: \"kubernetes.io/projected/a34254f0-0121-44f1-8ef3-27dda775a50d-kube-api-access-2s9jm\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.103675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.103567 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.103675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.103592 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34254f0-0121-44f1-8ef3-27dda775a50d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.108232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.108187 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-metrics-certs\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.108473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.108276 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.109372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.109349 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34254f0-0121-44f1-8ef3-27dda775a50d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.116995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.116962 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:59.120845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.120821 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9jm\" (UniqueName: \"kubernetes.io/projected/a34254f0-0121-44f1-8ef3-27dda775a50d-kube-api-access-2s9jm\") pod \"maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.216778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.216662 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:48:59.319280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.319206 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:48:59.558914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.558186 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9"] Oct 31 01:48:59.560921 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:48:59.560874 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34254f0_0121_44f1_8ef3_27dda775a50d.slice/crio-1e78391460b700760054dab3e4f84770fe807500760bd50576abcffaf6f6b60a WatchSource:0}: Error finding container 1e78391460b700760054dab3e4f84770fe807500760bd50576abcffaf6f6b60a: Status 404 returned error can't find the container with id 1e78391460b700760054dab3e4f84770fe807500760bd50576abcffaf6f6b60a Oct 31 01:48:59.755053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.755022 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b1b214-72d8-4350-b6f4-bd7026b12a76" path="/var/lib/kubelet/pods/87b1b214-72d8-4350-b6f4-bd7026b12a76/volumes" Oct 31 01:48:59.959562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:48:59.959432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:00.464220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:00.464194 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:49:00.538874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:00.538784 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" event={"ID":"a34254f0-0121-44f1-8ef3-27dda775a50d","Type":"ContainerStarted","Data":"f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf"} Oct 31 01:49:00.539134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:00.539111 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" event={"ID":"a34254f0-0121-44f1-8ef3-27dda775a50d","Type":"ContainerStarted","Data":"1e78391460b700760054dab3e4f84770fe807500760bd50576abcffaf6f6b60a"} Oct 31 01:49:00.539271 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:00.539250 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:49:00.561308 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:00.561232 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" podStartSLOduration=2.133272852 podStartE2EDuration="2.560555681s" totalImagesPullingTime="427.282829ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:00.560193879 +0000 UTC m=+1740.940439139" watchObservedRunningTime="2026-10-31 01:49:00.560555681 +0000 UTC m=+1740.940800905" Oct 31 01:49:00.667233 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:00.667206 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:01.537417 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:01.537383 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:02.344768 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:02.344684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:03.251844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:03.251806 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:03.444549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:03.444521 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:04.052738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:04.049919 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:04.642643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:04.642613 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:04.790738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:04.790021 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:04.793126 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:49:04.793091 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:49:05.578267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:05.578227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:05.742808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:05.742676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:05.903591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:05.903511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:06.109019 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:06.108979 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:06.437874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:06.437841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:06.565939 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:06.565908 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:49:06.742849 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:06.742820 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:06.921936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:06.921861 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:07.103808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:07.100434 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:07.256351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:07.256322 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:07.408113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:07.408030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:07.608741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:07.605454 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:07.974593 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:07.974547 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:08.504587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:08.504518 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:09.423322 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:09.423272 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:09.893285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:09.893256 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:10.032121 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:10.032090 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:10.564265 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:10.564229 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:11.158105 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:11.158068 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:11.303475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:11.303435 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:11.546410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:11.546377 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:12.413473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:12.413439 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:12.555906 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:12.555869 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:12.704535 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:12.704315 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:12.857048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:12.856961 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:13.072844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:13.072802 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:13.640249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:13.640215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:14.336577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:14.336543 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:14.472175 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:14.472149 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:14.697702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:14.697611 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:15.143463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:15.142954 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:15.544276 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:15.544244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:15.762487 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:15.762438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:16.059742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:16.051611 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:16.199173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:16.199142 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:16.639748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:16.639679 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:17.114394 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:17.114353 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:17.270597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:17.270562 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:17.777614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:17.777568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:18.457217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:18.457178 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:19.543754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.543686 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj"] Oct 31 01:49:19.550038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.550007 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.572736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.572238 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-073e24-serving-cert\"" Oct 31 01:49:19.576119 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.575080 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj"] Oct 31 01:49:19.638409 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.638378 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:19.675761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.675734 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wccx6\" (UniqueName: \"kubernetes.io/projected/5a31f1f8-fc98-4092-b13b-04c88f39caa7-kube-api-access-wccx6\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.675885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.675869 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.675924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.675911 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-metrics-certs\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.676006 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.675993 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31f1f8-fc98-4092-b13b-04c88f39caa7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.784372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.784337 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:19.791489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.786143 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31f1f8-fc98-4092-b13b-04c88f39caa7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.791489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.786978 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31f1f8-fc98-4092-b13b-04c88f39caa7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.791489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.790796 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wccx6\" (UniqueName: \"kubernetes.io/projected/5a31f1f8-fc98-4092-b13b-04c88f39caa7-kube-api-access-wccx6\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.791489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.790974 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.791489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.791032 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-metrics-certs\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.794967 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.794868 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-metrics-certs\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.794967 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.794868 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.803167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.803138 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wccx6\" (UniqueName: \"kubernetes.io/projected/5a31f1f8-fc98-4092-b13b-04c88f39caa7-kube-api-access-wccx6\") pod \"maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:19.882323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:19.882287 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:20.223110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:20.219370 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj"] Oct 31 01:49:20.311519 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:20.311485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:20.449587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:20.449561 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:20.667149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:20.666923 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:20.820386 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:20.820350 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" event={"ID":"5a31f1f8-fc98-4092-b13b-04c88f39caa7","Type":"ContainerStarted","Data":"9b69e6b51c90f3a6701fab678788842f60f0096c1ce40df367161801ea60d1de"} Oct 31 01:49:21.345509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:21.345475 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:21.638232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:21.638143 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:21.788190 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:21.788152 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:21.940802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:21.940431 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" event={"ID":"5a31f1f8-fc98-4092-b13b-04c88f39caa7","Type":"ContainerStarted","Data":"51ce78806a2053197a2858a6891605f7448776c7fb37097e4469e0fbd4549cfd"} Oct 31 01:49:21.941961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:21.941554 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:21.961564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:21.961507 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" podStartSLOduration=2.381182542 podStartE2EDuration="2.961488181s" totalImagesPullingTime="580.305639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:21.959940896 +0000 UTC m=+1762.340186152" watchObservedRunningTime="2026-10-31 01:49:21.961488181 +0000 UTC m=+1762.341733405" Oct 31 01:49:22.441023 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:22.440944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:23.248460 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:23.246035 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:23.888618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:23.888582 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:24.562261 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:24.562218 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:25.154054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:25.154013 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:25.838894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:25.838841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:26.764834 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:26.763279 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:27.430382 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:27.429422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:28.227249 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.227217 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh"] Oct 31 01:49:28.233582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.233553 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.278133 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.278063 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-676801-serving-cert\"" Oct 31 01:49:28.285026 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.285000 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh"] Oct 31 01:49:28.306985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.306952 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b140bf18-8776-4d39-bedf-3afed25d35e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.307239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.307034 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-metrics-certs\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.307313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.307231 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxmt\" (UniqueName: \"kubernetes.io/projected/b140bf18-8776-4d39-bedf-3afed25d35e6-kube-api-access-qnxmt\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.307383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.307362 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-maas-api-tls\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.375877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.375841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:28.409969 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.408532 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-maas-api-tls\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.409969 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.408645 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b140bf18-8776-4d39-bedf-3afed25d35e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.409969 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.408726 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-metrics-certs\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.409969 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.408886 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxmt\" (UniqueName: \"kubernetes.io/projected/b140bf18-8776-4d39-bedf-3afed25d35e6-kube-api-access-qnxmt\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.410475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.410453 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b140bf18-8776-4d39-bedf-3afed25d35e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.413668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.413645 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-maas-api-tls\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.413873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.413855 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-metrics-certs\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.420783 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.420703 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxmt\" (UniqueName: \"kubernetes.io/projected/b140bf18-8776-4d39-bedf-3afed25d35e6-kube-api-access-qnxmt\") pod \"maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.523945 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.523915 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:28.550037 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.550001 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:28.838630 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:49:28.838575 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb140bf18_8776_4d39_bedf_3afed25d35e6.slice/crio-3f36bf5dbc1fb335aa9bb3e6dd52d3a3ded695cca48d26d27bb8076865a80f03 WatchSource:0}: Error finding container 3f36bf5dbc1fb335aa9bb3e6dd52d3a3ded695cca48d26d27bb8076865a80f03: Status 404 returned error can't find the container with id 3f36bf5dbc1fb335aa9bb3e6dd52d3a3ded695cca48d26d27bb8076865a80f03 Oct 31 01:49:28.838801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.838687 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh"] Oct 31 01:49:28.983096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:28.983040 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:49:29.009320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:29.009282 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" event={"ID":"b140bf18-8776-4d39-bedf-3afed25d35e6","Type":"ContainerStarted","Data":"3f36bf5dbc1fb335aa9bb3e6dd52d3a3ded695cca48d26d27bb8076865a80f03"} Oct 31 01:49:29.134981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:29.134164 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:30.020155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:30.020103 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" event={"ID":"b140bf18-8776-4d39-bedf-3afed25d35e6","Type":"ContainerStarted","Data":"bde2e581acf9440fc1e3c3d47573f6764bfecc8ccf9661c59cc53324fc1b7ea6"} Oct 31 01:49:30.020640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:30.020186 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:30.047171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:30.047123 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" podStartSLOduration=1.6367921810000001 podStartE2EDuration="2.047107147s" totalImagesPullingTime="410.314966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:30.043305141 +0000 UTC m=+1770.423550365" watchObservedRunningTime="2026-10-31 01:49:30.047107147 +0000 UTC m=+1770.427352370" Oct 31 01:49:30.141345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:30.141315 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:30.983377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:30.983333 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:31.232889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:31.232857 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:31.417517 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:31.417485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:31.551318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:31.551283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:31.715803 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:31.715455 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:32.063371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:32.063333 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:32.865518 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:32.865486 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:33.549361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:33.549294 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:34.263294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:34.263240 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:34.789227 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:49:34.789189 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:49:34.965873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:34.965797 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:35.763760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:35.763659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:36.031922 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:36.031857 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:49:36.688647 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:36.688614 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:37.379096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:37.379057 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:38.353239 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:38.353198 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:39.267739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:39.256313 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:39.800108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:39.800076 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:39.940439 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:39.940404 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:40.315640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:40.315604 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:40.526171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:40.526138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:40.982985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:40.982953 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:41.201611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:41.201574 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:41.502482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:41.502443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:41.656741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:41.655106 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:42.355899 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:42.355866 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:43.241336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:43.241301 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:44.322311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:44.322239 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:44.942511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:44.942480 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:45.108583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:45.108497 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:45.276433 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:45.276364 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:45.952818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:45.952775 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:46.232914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:46.232825 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:46.370968 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:46.370922 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:46.963475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:46.963439 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:47.757973 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:47.753768 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:48.371769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:48.371707 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:49.208123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:49.208089 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:50.042697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:50.042663 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:50.368536 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:50.368456 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:50.878747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:50.877545 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:51.565695 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:51.565659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:52.161830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.161785 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw"] Oct 31 01:49:52.162310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.162278 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" podUID="ecc35581-a4a1-4ac2-9d93-f54ca34d378f" containerName="maas-api" containerID="cri-o://13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73" gracePeriod=30 Oct 31 01:49:52.252248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.248176 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm"] Oct 31 01:49:52.252248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.248465 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" podUID="7823fadd-da41-479a-87d3-83631ff40970" containerName="payload-processing" containerID="cri-o://167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70" gracePeriod=30 Oct 31 01:49:52.268676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.268619 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" podUID="7823fadd-da41-479a-87d3-83631ff40970" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.102:9004: connect: connection refused" Oct 31 01:49:52.270903 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.270868 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:52.406191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.406153 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:52.592740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.588387 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:52.741557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.741482 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:52.831296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.831270 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:49:52.887987 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.887938 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:52.895839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.895812 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:49:52.923777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.923744 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-openshift-service-ca\") pod \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " Oct 31 01:49:52.923960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.923844 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-kube-api-access-hlwlb\" (UniqueName: \"kubernetes.io/projected/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-kube-api-access-hlwlb\") pod \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " Oct 31 01:49:52.923960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.923876 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-metrics-certs\") pod \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " Oct 31 01:49:52.924104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.924000 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-maas-api-tls\") pod \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\" (UID: \"ecc35581-a4a1-4ac2-9d93-f54ca34d378f\") " Oct 31 01:49:52.924104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.924045 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7823fadd-da41-479a-87d3-83631ff40970-kube-api-access-kv5jx\" (UniqueName: \"kubernetes.io/projected/7823fadd-da41-479a-87d3-83631ff40970-kube-api-access-kv5jx\") pod \"7823fadd-da41-479a-87d3-83631ff40970\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " Oct 31 01:49:52.924224 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.924156 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7823fadd-da41-479a-87d3-83631ff40970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7823fadd-da41-479a-87d3-83631ff40970-plugins-config-volume\") pod \"7823fadd-da41-479a-87d3-83631ff40970\" (UID: \"7823fadd-da41-479a-87d3-83631ff40970\") " Oct 31 01:49:52.924755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.924153 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-openshift-service-ca" pod "ecc35581-a4a1-4ac2-9d93-f54ca34d378f" (UID: "ecc35581-a4a1-4ac2-9d93-f54ca34d378f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:49:52.925281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.925200 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7823fadd-da41-479a-87d3-83631ff40970-plugins-config-volume" pod "7823fadd-da41-479a-87d3-83631ff40970" (UID: "7823fadd-da41-479a-87d3-83631ff40970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:49:52.926777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.926695 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-kube-api-access-hlwlb" pod "ecc35581-a4a1-4ac2-9d93-f54ca34d378f" (UID: "ecc35581-a4a1-4ac2-9d93-f54ca34d378f"). InnerVolumeSpecName "kube-api-access-hlwlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:49:52.926960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.926940 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-metrics-certs" pod "ecc35581-a4a1-4ac2-9d93-f54ca34d378f" (UID: "ecc35581-a4a1-4ac2-9d93-f54ca34d378f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:49:52.927350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.927321 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7823fadd-da41-479a-87d3-83631ff40970-kube-api-access-kv5jx" pod "7823fadd-da41-479a-87d3-83631ff40970" (UID: "7823fadd-da41-479a-87d3-83631ff40970"). InnerVolumeSpecName "kube-api-access-kv5jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:49:52.927442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:52.927388 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-maas-api-tls" pod "ecc35581-a4a1-4ac2-9d93-f54ca34d378f" (UID: "ecc35581-a4a1-4ac2-9d93-f54ca34d378f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:49:53.027508 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.027471 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:49:53.027508 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.027507 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv5jx\" (UniqueName: \"kubernetes.io/projected/7823fadd-da41-479a-87d3-83631ff40970-kube-api-access-kv5jx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:49:53.027911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.027520 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7823fadd-da41-479a-87d3-83631ff40970-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:49:53.027911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.027529 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:49:53.027911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.027538 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlwlb\" (UniqueName: \"kubernetes.io/projected/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-kube-api-access-hlwlb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:49:53.027911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.027547 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecc35581-a4a1-4ac2-9d93-f54ca34d378f-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:49:53.263855 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.263817 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:53.289751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.289690 5090 generic.go:356] "Generic (PLEG): container finished" podID="ecc35581-a4a1-4ac2-9d93-f54ca34d378f" containerID="13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73" exitCode=0 Oct 31 01:49:53.289890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.289765 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" Oct 31 01:49:53.289890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.289780 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" event={"ID":"ecc35581-a4a1-4ac2-9d93-f54ca34d378f","Type":"ContainerDied","Data":"13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73"} Oct 31 01:49:53.289890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.289864 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw" event={"ID":"ecc35581-a4a1-4ac2-9d93-f54ca34d378f","Type":"ContainerDied","Data":"9b4d80f0dc89bdf2d3c6edf014c435ca801fb706d69a6ac54825237182d3341c"} Oct 31 01:49:53.290041 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.289896 5090 scope.go:122] "RemoveContainer" containerID="13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73" Oct 31 01:49:53.291420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.291403 5090 generic.go:356] "Generic (PLEG): container finished" podID="7823fadd-da41-479a-87d3-83631ff40970" containerID="167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70" exitCode=0 Oct 31 01:49:53.291507 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.291469 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" Oct 31 01:49:53.291570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.291495 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" event={"ID":"7823fadd-da41-479a-87d3-83631ff40970","Type":"ContainerDied","Data":"167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70"} Oct 31 01:49:53.291570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.291550 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm" event={"ID":"7823fadd-da41-479a-87d3-83631ff40970","Type":"ContainerDied","Data":"62c04025b12c6ce6f68e28c31326b10a1dee045502937a2a434a84798aaf6fbe"} Oct 31 01:49:53.303760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.303735 5090 scope.go:122] "RemoveContainer" containerID="13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73" Oct 31 01:49:53.304313 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:49:53.304291 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73\": container with ID starting with 13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73 not found: ID does not exist" containerID="13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73" Oct 31 01:49:53.304483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.304440 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73"} err="failed to get container status \"13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73\": rpc error: code = NotFound desc = could not find container \"13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73\": container with ID starting with 13bc2d669d47d0927f6f989e9e21eeeb3806eaa098e75c64520a3a5bcb4e0f73 not found: ID does not exist" Oct 31 01:49:53.304483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.304481 5090 scope.go:122] "RemoveContainer" containerID="167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70" Oct 31 01:49:53.314370 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.314351 5090 scope.go:122] "RemoveContainer" containerID="167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70" Oct 31 01:49:53.314615 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:49:53.314599 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70\": container with ID starting with 167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70 not found: ID does not exist" containerID="167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70" Oct 31 01:49:53.314663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.314621 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70"} err="failed to get container status \"167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70\": rpc error: code = NotFound desc = could not find container \"167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70\": container with ID starting with 167156e8c6de15378cdc6ef97b2e961a4ed6bd787245012d57c01c89b390bb70 not found: ID does not exist" Oct 31 01:49:53.321806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.321785 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw"] Oct 31 01:49:53.326977 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.326957 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9f0694-767f67976-vbnfw"] Oct 31 01:49:53.338389 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.338367 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm"] Oct 31 01:49:53.342858 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.342839 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9f0694-58c7447fd-sqmcm"] Oct 31 01:49:53.548103 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.548061 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg"] Oct 31 01:49:53.550353 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.549625 5090 cpu_manager.go:414] "containerMap: removing container" podUID="ecc35581-a4a1-4ac2-9d93-f54ca34d378f" containerName="maas-api" Oct 31 01:49:53.550353 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.549652 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc35581-a4a1-4ac2-9d93-f54ca34d378f" containerName="maas-api" Oct 31 01:49:53.550560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.550423 5090 cpu_manager.go:414] "containerMap: removing container" podUID="7823fadd-da41-479a-87d3-83631ff40970" containerName="payload-processing" Oct 31 01:49:53.550560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.550436 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7823fadd-da41-479a-87d3-83631ff40970" containerName="payload-processing" Oct 31 01:49:53.550678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.550604 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc35581-a4a1-4ac2-9d93-f54ca34d378f" containerName="maas-api" Oct 31 01:49:53.550678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.550618 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="7823fadd-da41-479a-87d3-83631ff40970" containerName="payload-processing" Oct 31 01:49:53.556914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.556885 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.563191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.563162 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg"] Oct 31 01:49:53.573961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.572993 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e52ba1-serving-cert\"" Oct 31 01:49:53.638820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.638790 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhr2r\" (UniqueName: \"kubernetes.io/projected/9dbf4908-d08b-4b67-9a74-c15201706548-kube-api-access-mhr2r\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.639223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.638869 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-maas-api-tls\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.639223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.638973 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-metrics-certs\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.639223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.639045 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbf4908-d08b-4b67-9a74-c15201706548-openshift-service-ca\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.640140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.640117 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:53.742443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.741293 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhr2r\" (UniqueName: \"kubernetes.io/projected/9dbf4908-d08b-4b67-9a74-c15201706548-kube-api-access-mhr2r\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.742443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.741396 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-maas-api-tls\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.742443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.741469 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-metrics-certs\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.742443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.741553 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbf4908-d08b-4b67-9a74-c15201706548-openshift-service-ca\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.749747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.742504 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbf4908-d08b-4b67-9a74-c15201706548-openshift-service-ca\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.755481 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.754900 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-maas-api-tls\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.756787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.756555 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-metrics-certs\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.756787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.756668 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhr2r\" (UniqueName: \"kubernetes.io/projected/9dbf4908-d08b-4b67-9a74-c15201706548-kube-api-access-mhr2r\") pod \"maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:53.765003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.764978 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7823fadd-da41-479a-87d3-83631ff40970" path="/var/lib/kubelet/pods/7823fadd-da41-479a-87d3-83631ff40970/volumes" Oct 31 01:49:53.765502 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.765483 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc35581-a4a1-4ac2-9d93-f54ca34d378f" path="/var/lib/kubelet/pods/ecc35581-a4a1-4ac2-9d93-f54ca34d378f/volumes" Oct 31 01:49:53.780241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.780220 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:53.877609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:53.877518 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:54.191039 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:54.191011 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg"] Oct 31 01:49:54.191165 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:49:54.191014 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbf4908_d08b_4b67_9a74_c15201706548.slice/crio-ca898f7c6206a1d7c5d9d16a1c50246c83e9a88fb460e6e1dcee0abac579908c WatchSource:0}: Error finding container ca898f7c6206a1d7c5d9d16a1c50246c83e9a88fb460e6e1dcee0abac579908c: Status 404 returned error can't find the container with id ca898f7c6206a1d7c5d9d16a1c50246c83e9a88fb460e6e1dcee0abac579908c Oct 31 01:49:54.300911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:54.300875 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" event={"ID":"9dbf4908-d08b-4b67-9a74-c15201706548","Type":"ContainerStarted","Data":"ca898f7c6206a1d7c5d9d16a1c50246c83e9a88fb460e6e1dcee0abac579908c"} Oct 31 01:49:54.448639 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:54.448566 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:55.310000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:55.309962 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" event={"ID":"9dbf4908-d08b-4b67-9a74-c15201706548","Type":"ContainerStarted","Data":"b4b1cc52e443392b14f733811fa0d476c90e5947caae7d7131ffb1087803577a"} Oct 31 01:49:55.310478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:55.310128 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:49:55.330400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:55.330355 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" podStartSLOduration=1.760438572 podStartE2EDuration="2.330340998s" totalImagesPullingTime="569.902426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:49:55.328016726 +0000 UTC m=+1795.708261976" watchObservedRunningTime="2026-10-31 01:49:55.330340998 +0000 UTC m=+1795.710586220" Oct 31 01:49:55.375500 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:55.375470 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:56.115471 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:56.115443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:56.948421 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:56.948388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:57.954648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:57.954616 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:58.480385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:58.480352 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:58.773406 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:58.773371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:58.930480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:58.930442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:59.076102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:59.076014 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:49:59.846158 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:49:59.846113 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:00.132476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:00.132320 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:50:00.134631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:00.134605 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:50:00.138171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:00.138148 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:50:00.140902 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:00.140878 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:50:00.576798 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:00.575362 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:00.837324 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:00.837249 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:00.983660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:00.983625 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:01.277737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:01.271437 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:01.341162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:01.341134 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:50:01.742863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:01.742832 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:02.131259 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:02.131169 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:02.294304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:02.294270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:02.658078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:02.657927 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:03.475836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:03.475801 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:03.683021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:03.682986 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:03.889774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:03.889740 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:04.666797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:04.666762 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:04.789109 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:50:04.789074 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:50:05.492694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:05.492654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:06.345675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:06.345638 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:07.021084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:07.021053 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:07.732887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:07.732852 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:08.544600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:08.544565 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:09.252016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:09.251971 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:10.040150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:10.040116 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:10.230746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:10.230683 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:10.683593 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:10.683556 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:11.364366 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.364325 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz"] Oct 31 01:50:11.371766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.371741 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.383505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.383476 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz"] Oct 31 01:50:11.383824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.383804 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-90c4d1-serving-cert\"" Oct 31 01:50:11.395185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.393617 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqd6\" (UniqueName: \"kubernetes.io/projected/4223400f-40ee-4acc-b758-cb23412a7724-kube-api-access-chqd6\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.395185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.393770 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.395185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.393893 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4223400f-40ee-4acc-b758-cb23412a7724-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.395185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.393991 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-metrics-certs\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.455016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.454982 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:11.495676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.495637 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.495920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.495768 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4223400f-40ee-4acc-b758-cb23412a7724-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.495920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.495809 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-metrics-certs\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.495920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.495895 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqd6\" (UniqueName: \"kubernetes.io/projected/4223400f-40ee-4acc-b758-cb23412a7724-kube-api-access-chqd6\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.496557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.496532 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4223400f-40ee-4acc-b758-cb23412a7724-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.498479 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.498456 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-metrics-certs\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.498568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.498542 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-maas-api-tls\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.559665 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.552050 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqd6\" (UniqueName: \"kubernetes.io/projected/4223400f-40ee-4acc-b758-cb23412a7724-kube-api-access-chqd6\") pod \"maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.602425 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.602391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:11.701914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.701824 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:11.987747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:11.987697 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz"] Oct 31 01:50:11.991104 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:50:11.991070 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4223400f_40ee_4acc_b758_cb23412a7724.slice/crio-1ea49eef558d1bc5dfdb2a2e0b45229ffd3995208ae19ab3542d8eaf1a1426e8 WatchSource:0}: Error finding container 1ea49eef558d1bc5dfdb2a2e0b45229ffd3995208ae19ab3542d8eaf1a1426e8: Status 404 returned error can't find the container with id 1ea49eef558d1bc5dfdb2a2e0b45229ffd3995208ae19ab3542d8eaf1a1426e8 Oct 31 01:50:12.472016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:12.469509 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:12.506256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:12.506226 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" event={"ID":"4223400f-40ee-4acc-b758-cb23412a7724","Type":"ContainerStarted","Data":"1ea49eef558d1bc5dfdb2a2e0b45229ffd3995208ae19ab3542d8eaf1a1426e8"} Oct 31 01:50:13.263019 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:13.259378 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:13.514280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:13.514176 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" event={"ID":"4223400f-40ee-4acc-b758-cb23412a7724","Type":"ContainerStarted","Data":"1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82"} Oct 31 01:50:13.514280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:13.514241 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:13.533670 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:13.533604 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" podStartSLOduration=2.046308731 podStartE2EDuration="2.533586145s" totalImagesPullingTime="487.277414ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:13.532488901 +0000 UTC m=+1813.912734144" watchObservedRunningTime="2026-10-31 01:50:13.533586145 +0000 UTC m=+1813.913831369" Oct 31 01:50:14.050468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:14.050431 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:14.855318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:14.851220 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:15.752060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:15.751977 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:16.512615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:16.512581 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:17.277763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:17.277732 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:17.727630 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:17.727548 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9"] Oct 31 01:50:17.728454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:17.728421 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" podUID="a34254f0-0121-44f1-8ef3-27dda775a50d" containerName="maas-api" containerID="cri-o://f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf" gracePeriod=30 Oct 31 01:50:17.833381 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:17.833343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:17.981619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:17.981478 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:18.133224 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.133134 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:18.281401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.281362 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:18.284761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.284729 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:50:18.327285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.327250 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34254f0-0121-44f1-8ef3-27dda775a50d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34254f0-0121-44f1-8ef3-27dda775a50d-openshift-service-ca\") pod \"a34254f0-0121-44f1-8ef3-27dda775a50d\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " Oct 31 01:50:18.327450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.327320 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34254f0-0121-44f1-8ef3-27dda775a50d-kube-api-access-2s9jm\" (UniqueName: \"kubernetes.io/projected/a34254f0-0121-44f1-8ef3-27dda775a50d-kube-api-access-2s9jm\") pod \"a34254f0-0121-44f1-8ef3-27dda775a50d\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " Oct 31 01:50:18.327450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.327410 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-maas-api-tls\") pod \"a34254f0-0121-44f1-8ef3-27dda775a50d\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " Oct 31 01:50:18.327589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.327452 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-metrics-certs\") pod \"a34254f0-0121-44f1-8ef3-27dda775a50d\" (UID: \"a34254f0-0121-44f1-8ef3-27dda775a50d\") " Oct 31 01:50:18.327701 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.327675 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34254f0-0121-44f1-8ef3-27dda775a50d-openshift-service-ca" pod "a34254f0-0121-44f1-8ef3-27dda775a50d" (UID: "a34254f0-0121-44f1-8ef3-27dda775a50d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:50:18.328143 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.328114 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34254f0-0121-44f1-8ef3-27dda775a50d-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:18.330703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.330669 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-maas-api-tls" pod "a34254f0-0121-44f1-8ef3-27dda775a50d" (UID: "a34254f0-0121-44f1-8ef3-27dda775a50d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:50:18.332562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.332539 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-metrics-certs" pod "a34254f0-0121-44f1-8ef3-27dda775a50d" (UID: "a34254f0-0121-44f1-8ef3-27dda775a50d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:50:18.333196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.333152 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34254f0-0121-44f1-8ef3-27dda775a50d-kube-api-access-2s9jm" pod "a34254f0-0121-44f1-8ef3-27dda775a50d" (UID: "a34254f0-0121-44f1-8ef3-27dda775a50d"). InnerVolumeSpecName "kube-api-access-2s9jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:50:18.431838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.431753 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s9jm\" (UniqueName: \"kubernetes.io/projected/a34254f0-0121-44f1-8ef3-27dda775a50d-kube-api-access-2s9jm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:18.431838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.431790 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:18.431838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.431807 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34254f0-0121-44f1-8ef3-27dda775a50d-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:18.434408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.432241 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:18.599217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.599116 5090 generic.go:356] "Generic (PLEG): container finished" podID="a34254f0-0121-44f1-8ef3-27dda775a50d" containerID="f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf" exitCode=0 Oct 31 01:50:18.599217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.599166 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" Oct 31 01:50:18.599217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.599207 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" event={"ID":"a34254f0-0121-44f1-8ef3-27dda775a50d","Type":"ContainerDied","Data":"f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf"} Oct 31 01:50:18.599494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.599267 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9" event={"ID":"a34254f0-0121-44f1-8ef3-27dda775a50d","Type":"ContainerDied","Data":"1e78391460b700760054dab3e4f84770fe807500760bd50576abcffaf6f6b60a"} Oct 31 01:50:18.599494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.599284 5090 scope.go:122] "RemoveContainer" containerID="f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf" Oct 31 01:50:18.613614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.613594 5090 scope.go:122] "RemoveContainer" containerID="f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf" Oct 31 01:50:18.613994 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:50:18.613969 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf\": container with ID starting with f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf not found: ID does not exist" containerID="f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf" Oct 31 01:50:18.614083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.614011 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf"} err="failed to get container status \"f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf\": rpc error: code = NotFound desc = could not find container \"f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf\": container with ID starting with f775a39f4e32cab7145b3282e1937c0a1d3bcc0f70f56a55478eec62779bd7bf not found: ID does not exist" Oct 31 01:50:18.631395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.631366 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9"] Oct 31 01:50:18.634727 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.634695 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cbbaec-b4d59898c-bj5v9"] Oct 31 01:50:18.898117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:18.898026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:19.275126 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:19.274696 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:19.561983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:19.561917 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:50:19.747478 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:19.747446 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34254f0-0121-44f1-8ef3-27dda775a50d" path="/var/lib/kubelet/pods/a34254f0-0121-44f1-8ef3-27dda775a50d/volumes" Oct 31 01:50:20.034336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:20.034289 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:20.684209 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:20.684178 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:21.467707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:21.467667 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:21.646941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:21.646903 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:21.786942 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:21.786905 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:22.528728 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:22.520525 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:23.377802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:23.377741 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:24.340390 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:24.340339 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:25.059743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:25.059461 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:25.482358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:25.482276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:25.620252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:25.620221 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:25.775624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:25.774939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:25.935326 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:25.933937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:26.081777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:26.081617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:26.406968 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:26.406872 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:26.638951 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:26.638919 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:26.823097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:26.823061 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:27.494989 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:27.494935 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:27.655203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:27.655174 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:27.875160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:27.875016 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:28.157605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:28.157498 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:28.312740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:28.312535 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:28.456620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:28.455809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:29.272758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:29.272699 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:30.235997 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:30.231807 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:31.130008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:31.129979 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:31.357898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:31.342217 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:31.985787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:31.985749 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:32.182267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:32.182196 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:32.683638 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:32.683605 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:32.823122 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:32.823091 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:33.657549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:33.657504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:33.871986 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:33.871959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:34.139301 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:34.139272 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:34.765970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:34.765936 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:34.804706 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:50:34.804669 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:50:35.444408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:35.444375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:36.164638 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:36.164607 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:36.841003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:36.839386 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:37.448838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:37.448779 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:38.133850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:38.133817 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:38.725738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:38.725698 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:39.294003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.286694 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh"] Oct 31 01:50:39.294003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.286950 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" podUID="b140bf18-8776-4d39-bedf-3afed25d35e6" containerName="maas-api" containerID="cri-o://bde2e581acf9440fc1e3c3d47573f6764bfecc8ccf9661c59cc53324fc1b7ea6" gracePeriod=30 Oct 31 01:50:39.372165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.372126 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:39.718232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.711211 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:39.793475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.793426 5090 generic.go:356] "Generic (PLEG): container finished" podID="b140bf18-8776-4d39-bedf-3afed25d35e6" containerID="bde2e581acf9440fc1e3c3d47573f6764bfecc8ccf9661c59cc53324fc1b7ea6" exitCode=0 Oct 31 01:50:39.793952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.793777 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" event={"ID":"b140bf18-8776-4d39-bedf-3afed25d35e6","Type":"ContainerDied","Data":"bde2e581acf9440fc1e3c3d47573f6764bfecc8ccf9661c59cc53324fc1b7ea6"} Oct 31 01:50:39.793952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.793847 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" event={"ID":"b140bf18-8776-4d39-bedf-3afed25d35e6","Type":"ContainerDied","Data":"3f36bf5dbc1fb335aa9bb3e6dd52d3a3ded695cca48d26d27bb8076865a80f03"} Oct 31 01:50:39.793952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.793873 5090 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f36bf5dbc1fb335aa9bb3e6dd52d3a3ded695cca48d26d27bb8076865a80f03" Oct 31 01:50:39.797837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.797769 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v"] Oct 31 01:50:39.799157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.799128 5090 cpu_manager.go:414] "containerMap: removing container" podUID="a34254f0-0121-44f1-8ef3-27dda775a50d" containerName="maas-api" Oct 31 01:50:39.799306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.799292 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34254f0-0121-44f1-8ef3-27dda775a50d" containerName="maas-api" Oct 31 01:50:39.799590 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.799575 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34254f0-0121-44f1-8ef3-27dda775a50d" containerName="maas-api" Oct 31 01:50:39.804005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.803975 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:39.806684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.804792 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:50:39.816593 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.816537 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-a779b3-serving-cert\"" Oct 31 01:50:39.821577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.821552 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v"] Oct 31 01:50:39.868589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:39.868557 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:40.004129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.004050 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-metrics-certs\") pod \"b140bf18-8776-4d39-bedf-3afed25d35e6\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " Oct 31 01:50:40.004558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.004461 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b140bf18-8776-4d39-bedf-3afed25d35e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b140bf18-8776-4d39-bedf-3afed25d35e6-openshift-service-ca\") pod \"b140bf18-8776-4d39-bedf-3afed25d35e6\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " Oct 31 01:50:40.004558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.004521 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-maas-api-tls\") pod \"b140bf18-8776-4d39-bedf-3afed25d35e6\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " Oct 31 01:50:40.005106 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.005060 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b140bf18-8776-4d39-bedf-3afed25d35e6-kube-api-access-qnxmt\" (UniqueName: \"kubernetes.io/projected/b140bf18-8776-4d39-bedf-3afed25d35e6-kube-api-access-qnxmt\") pod \"b140bf18-8776-4d39-bedf-3afed25d35e6\" (UID: \"b140bf18-8776-4d39-bedf-3afed25d35e6\") " Oct 31 01:50:40.005678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.005643 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab0118-dff0-4392-b6c4-06c7d7619070-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.005942 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.005888 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.006163 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.006085 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scp7v\" (UniqueName: \"kubernetes.io/projected/4cab0118-dff0-4392-b6c4-06c7d7619070-kube-api-access-scp7v\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.006544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.006295 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-metrics-certs\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.009690 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.009374 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b140bf18-8776-4d39-bedf-3afed25d35e6-openshift-service-ca" pod "b140bf18-8776-4d39-bedf-3afed25d35e6" (UID: "b140bf18-8776-4d39-bedf-3afed25d35e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:50:40.017221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.017189 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:40.033081 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.033045 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b140bf18-8776-4d39-bedf-3afed25d35e6-kube-api-access-qnxmt" pod "b140bf18-8776-4d39-bedf-3afed25d35e6" (UID: "b140bf18-8776-4d39-bedf-3afed25d35e6"). InnerVolumeSpecName "kube-api-access-qnxmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:50:40.033475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.033376 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-metrics-certs" pod "b140bf18-8776-4d39-bedf-3afed25d35e6" (UID: "b140bf18-8776-4d39-bedf-3afed25d35e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:50:40.033792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.033654 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-maas-api-tls" pod "b140bf18-8776-4d39-bedf-3afed25d35e6" (UID: "b140bf18-8776-4d39-bedf-3afed25d35e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:50:40.108101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.107693 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab0118-dff0-4392-b6c4-06c7d7619070-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.108101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.107814 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.108101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.107889 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scp7v\" (UniqueName: \"kubernetes.io/projected/4cab0118-dff0-4392-b6c4-06c7d7619070-kube-api-access-scp7v\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.108101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.107971 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-metrics-certs\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.108671 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.108637 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab0118-dff0-4392-b6c4-06c7d7619070-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.108791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.108643 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxmt\" (UniqueName: \"kubernetes.io/projected/b140bf18-8776-4d39-bedf-3afed25d35e6-kube-api-access-qnxmt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:40.108791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.108700 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:40.108791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.108764 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b140bf18-8776-4d39-bedf-3afed25d35e6-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:40.108791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.108783 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b140bf18-8776-4d39-bedf-3afed25d35e6-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:50:40.112156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.112132 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-metrics-certs\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.112781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.112706 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-maas-api-tls\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.120374 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.120353 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scp7v\" (UniqueName: \"kubernetes.io/projected/4cab0118-dff0-4392-b6c4-06c7d7619070-kube-api-access-scp7v\") pod \"maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.125258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.125234 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:40.399352 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.399225 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:40.431865 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.431834 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v"] Oct 31 01:50:40.487055 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:50:40.479515 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cab0118_dff0_4392_b6c4_06c7d7619070.slice/crio-4f2350ee01cb274dc7400de9dc6cb6fa7b38e05ad00dc67ec185c8bcb41c6bb9 WatchSource:0}: Error finding container 4f2350ee01cb274dc7400de9dc6cb6fa7b38e05ad00dc67ec185c8bcb41c6bb9: Status 404 returned error can't find the container with id 4f2350ee01cb274dc7400de9dc6cb6fa7b38e05ad00dc67ec185c8bcb41c6bb9 Oct 31 01:50:40.801981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.801943 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" event={"ID":"4cab0118-dff0-4392-b6c4-06c7d7619070","Type":"ContainerStarted","Data":"4f2350ee01cb274dc7400de9dc6cb6fa7b38e05ad00dc67ec185c8bcb41c6bb9"} Oct 31 01:50:40.802486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.802100 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh" Oct 31 01:50:40.826498 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.826474 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh"] Oct 31 01:50:40.834566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.834545 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-676801-6fbf67bf59-7sfmh"] Oct 31 01:50:40.894752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:40.886316 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:41.559318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.559285 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw"] Oct 31 01:50:41.560058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.560036 5090 cpu_manager.go:414] "containerMap: removing container" podUID="b140bf18-8776-4d39-bedf-3afed25d35e6" containerName="maas-api" Oct 31 01:50:41.560058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.560056 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b140bf18-8776-4d39-bedf-3afed25d35e6" containerName="maas-api" Oct 31 01:50:41.560265 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.560183 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="b140bf18-8776-4d39-bedf-3afed25d35e6" containerName="maas-api" Oct 31 01:50:41.563386 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.563367 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.572262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.572234 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-611693-serving-cert\"" Oct 31 01:50:41.578931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.578572 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw"] Oct 31 01:50:41.649552 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.649521 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a699d3-68ab-434a-8c40-de3b0afde7b8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.649697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.649572 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8jt\" (UniqueName: \"kubernetes.io/projected/e6a699d3-68ab-434a-8c40-de3b0afde7b8-kube-api-access-vm8jt\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.649697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.649654 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-metrics-certs\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.649842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.649731 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.660412 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.660388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:41.747471 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.747447 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b140bf18-8776-4d39-bedf-3afed25d35e6" path="/var/lib/kubelet/pods/b140bf18-8776-4d39-bedf-3afed25d35e6/volumes" Oct 31 01:50:41.750868 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.750846 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a699d3-68ab-434a-8c40-de3b0afde7b8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.750957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.750884 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8jt\" (UniqueName: \"kubernetes.io/projected/e6a699d3-68ab-434a-8c40-de3b0afde7b8-kube-api-access-vm8jt\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.750957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.750914 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-metrics-certs\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.750957 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.750943 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.751563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.751539 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a699d3-68ab-434a-8c40-de3b0afde7b8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.753492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.753472 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.753569 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.753534 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-metrics-certs\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.762817 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.762799 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8jt\" (UniqueName: \"kubernetes.io/projected/e6a699d3-68ab-434a-8c40-de3b0afde7b8-kube-api-access-vm8jt\") pod \"maas-api-e2e-praxis-a-611693-5659748b96-4dqlw\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:41.825675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.825575 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:41.828545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.827874 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" event={"ID":"4cab0118-dff0-4392-b6c4-06c7d7619070","Type":"ContainerStarted","Data":"102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e"} Oct 31 01:50:41.828545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.828112 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:41.849686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.849644 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" podStartSLOduration=2.404530123 podStartE2EDuration="2.84962989s" totalImagesPullingTime="445.099767ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:41.846816549 +0000 UTC m=+1842.227061773" watchObservedRunningTime="2026-10-31 01:50:41.84962989 +0000 UTC m=+1842.229875112" Oct 31 01:50:41.883404 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:41.883368 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:42.257388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:42.257356 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw"] Oct 31 01:50:42.521251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:42.521154 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:42.835920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:42.835891 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" event={"ID":"e6a699d3-68ab-434a-8c40-de3b0afde7b8","Type":"ContainerStarted","Data":"c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d"} Oct 31 01:50:42.836285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:42.835948 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:42.836285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:42.835970 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" event={"ID":"e6a699d3-68ab-434a-8c40-de3b0afde7b8","Type":"ContainerStarted","Data":"0fe65e952849bdb0044652fb5b56baff5604b61002b49449e8b4ea35f8852d17"} Oct 31 01:50:42.860839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:42.860053 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" podStartSLOduration=1.397371922 podStartE2EDuration="1.860037026s" totalImagesPullingTime="462.665104ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:50:42.855627155 +0000 UTC m=+1843.235872378" watchObservedRunningTime="2026-10-31 01:50:42.860037026 +0000 UTC m=+1843.240282250" Oct 31 01:50:43.665740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:43.658466 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:44.530185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:44.530145 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:45.416791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:45.416756 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:46.005286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:46.005248 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:46.705242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:46.705212 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:47.298938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:47.298903 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:47.477770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:47.477688 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:47.632917 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:47.632837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:47.791687 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:47.791647 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:47.857205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:47.857177 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:50:48.322702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:48.322663 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:48.849468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:48.849440 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:50:49.259794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:49.248255 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:50.058659 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:50.058615 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:50.422000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:50.421921 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:51.016897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:51.016858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:51.862631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:51.859526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:53.064089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:53.064051 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:53.887258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:53.887219 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:55.312686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:55.312623 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:56.228348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:56.228316 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:56.425527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:56.425495 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:56.934035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:56.933894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:57.366329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:57.366231 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:57.656099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:57.656012 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:57.801710 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:57.801682 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:58.243540 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:58.243510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:58.387982 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:58.387950 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:58.672796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:58.672654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:58.810962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:58.810930 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:58.970864 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:58.968568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:59.339880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:59.339511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:59.513320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:59.513275 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:50:59.959344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:50:59.959292 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:00.867337 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:00.867303 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:01.752907 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:01.752867 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:01.979374 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:01.979348 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:02.830228 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:02.830186 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:03.033741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:03.033539 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:03.207391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:03.207306 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:03.419746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:03.402628 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:03.615136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:03.615103 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj"] Oct 31 01:51:03.616134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:03.616094 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" podUID="5a31f1f8-fc98-4092-b13b-04c88f39caa7" containerName="maas-api" containerID="cri-o://51ce78806a2053197a2858a6891605f7448776c7fb37097e4469e0fbd4549cfd" gracePeriod=30 Oct 31 01:51:03.702704 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:03.702581 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:03.848246 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:03.848209 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:04.085018 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.083747 5090 generic.go:356] "Generic (PLEG): container finished" podID="5a31f1f8-fc98-4092-b13b-04c88f39caa7" containerID="51ce78806a2053197a2858a6891605f7448776c7fb37097e4469e0fbd4549cfd" exitCode=0 Oct 31 01:51:04.085018 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.083847 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" event={"ID":"5a31f1f8-fc98-4092-b13b-04c88f39caa7","Type":"ContainerDied","Data":"51ce78806a2053197a2858a6891605f7448776c7fb37097e4469e0fbd4549cfd"} Oct 31 01:51:04.105160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.105135 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:51:04.165151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.165118 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:04.256247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.256033 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-maas-api-tls\") pod \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " Oct 31 01:51:04.256247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.256170 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a31f1f8-fc98-4092-b13b-04c88f39caa7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31f1f8-fc98-4092-b13b-04c88f39caa7-openshift-service-ca\") pod \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " Oct 31 01:51:04.256692 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.256660 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-metrics-certs\") pod \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " Oct 31 01:51:04.256859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.256757 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a31f1f8-fc98-4092-b13b-04c88f39caa7-kube-api-access-wccx6\" (UniqueName: \"kubernetes.io/projected/5a31f1f8-fc98-4092-b13b-04c88f39caa7-kube-api-access-wccx6\") pod \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\" (UID: \"5a31f1f8-fc98-4092-b13b-04c88f39caa7\") " Oct 31 01:51:04.257193 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.257166 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a31f1f8-fc98-4092-b13b-04c88f39caa7-openshift-service-ca" pod "5a31f1f8-fc98-4092-b13b-04c88f39caa7" (UID: "5a31f1f8-fc98-4092-b13b-04c88f39caa7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:51:04.258060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.258037 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31f1f8-fc98-4092-b13b-04c88f39caa7-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:04.259969 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.259943 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-metrics-certs" pod "5a31f1f8-fc98-4092-b13b-04c88f39caa7" (UID: "5a31f1f8-fc98-4092-b13b-04c88f39caa7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:04.268658 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.268627 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a31f1f8-fc98-4092-b13b-04c88f39caa7-kube-api-access-wccx6" pod "5a31f1f8-fc98-4092-b13b-04c88f39caa7" (UID: "5a31f1f8-fc98-4092-b13b-04c88f39caa7"). InnerVolumeSpecName "kube-api-access-wccx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:51:04.270161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.270134 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-maas-api-tls" pod "5a31f1f8-fc98-4092-b13b-04c88f39caa7" (UID: "5a31f1f8-fc98-4092-b13b-04c88f39caa7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:04.348816 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.348644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:04.360202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.360123 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:04.360202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.360160 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wccx6\" (UniqueName: \"kubernetes.io/projected/5a31f1f8-fc98-4092-b13b-04c88f39caa7-kube-api-access-wccx6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:04.360202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.360178 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31f1f8-fc98-4092-b13b-04c88f39caa7-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:04.663787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.658689 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:04.818479 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:51:04.818423 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:51:04.825701 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:04.825510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:05.091845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:05.091810 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" event={"ID":"5a31f1f8-fc98-4092-b13b-04c88f39caa7","Type":"ContainerDied","Data":"9b69e6b51c90f3a6701fab678788842f60f0096c1ce40df367161801ea60d1de"} Oct 31 01:51:05.091845 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:05.091841 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" Oct 31 01:51:05.092379 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:05.091875 5090 scope.go:122] "RemoveContainer" containerID="51ce78806a2053197a2858a6891605f7448776c7fb37097e4469e0fbd4549cfd" Oct 31 01:51:05.119217 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:05.119191 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj"] Oct 31 01:51:05.122890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:05.122866 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj"] Oct 31 01:51:05.469621 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:05.467540 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:05.748351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:05.748319 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a31f1f8-fc98-4092-b13b-04c88f39caa7" path="/var/lib/kubelet/pods/5a31f1f8-fc98-4092-b13b-04c88f39caa7/volumes" Oct 31 01:51:06.473401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:06.473294 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:06.958583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:06.958525 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-073e24-76b9d676d7-m7vjj" podUID="5a31f1f8-fc98-4092-b13b-04c88f39caa7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.105:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 01:51:07.380110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:07.378582 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:08.383560 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:08.383521 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:09.067070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:09.067014 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:09.892596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:09.892528 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:10.543087 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:10.543034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:11.155763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.155733 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p"] Oct 31 01:51:11.156395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.156381 5090 cpu_manager.go:414] "containerMap: removing container" podUID="5a31f1f8-fc98-4092-b13b-04c88f39caa7" containerName="maas-api" Oct 31 01:51:11.156442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.156397 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a31f1f8-fc98-4092-b13b-04c88f39caa7" containerName="maas-api" Oct 31 01:51:11.156521 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.156511 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a31f1f8-fc98-4092-b13b-04c88f39caa7" containerName="maas-api" Oct 31 01:51:11.161565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.161547 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.168251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.168224 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-726dd1-serving-cert\"" Oct 31 01:51:11.169209 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.169188 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p"] Oct 31 01:51:11.227692 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.227642 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:11.299097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.299059 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eec0817-1add-4a0b-af21-3cbdf992c01a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.299296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.299121 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.299390 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.299363 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-metrics-certs\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.299460 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.299425 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhr9x\" (UniqueName: \"kubernetes.io/projected/4eec0817-1add-4a0b-af21-3cbdf992c01a-kube-api-access-zhr9x\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.401299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.401264 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eec0817-1add-4a0b-af21-3cbdf992c01a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.401476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.401333 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.401564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.401529 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-metrics-certs\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.401564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.401556 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhr9x\" (UniqueName: \"kubernetes.io/projected/4eec0817-1add-4a0b-af21-3cbdf992c01a-kube-api-access-zhr9x\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.402107 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.402075 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eec0817-1add-4a0b-af21-3cbdf992c01a-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.404267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.404241 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-maas-api-tls\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.404476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.404460 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-metrics-certs\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.418380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.418305 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhr9x\" (UniqueName: \"kubernetes.io/projected/4eec0817-1add-4a0b-af21-3cbdf992c01a-kube-api-access-zhr9x\") pod \"maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.493027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.492996 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:11.512529 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.512468 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:11.781941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.781910 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:11.784542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:11.784516 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p"] Oct 31 01:51:11.786946 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:51:11.786919 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eec0817_1add_4a0b_af21_3cbdf992c01a.slice/crio-22d45b202ab5b31617cca067fb3e229ad771d2b98fd335f7221cb5947013a654 WatchSource:0}: Error finding container 22d45b202ab5b31617cca067fb3e229ad771d2b98fd335f7221cb5947013a654: Status 404 returned error can't find the container with id 22d45b202ab5b31617cca067fb3e229ad771d2b98fd335f7221cb5947013a654 Oct 31 01:51:12.155524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:12.155433 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:12.251819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:12.251780 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" event={"ID":"4eec0817-1add-4a0b-af21-3cbdf992c01a","Type":"ContainerStarted","Data":"22d45b202ab5b31617cca067fb3e229ad771d2b98fd335f7221cb5947013a654"} Oct 31 01:51:12.349549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:12.349520 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:12.579651 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:12.579617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:12.990651 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:12.990558 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:13.157242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:13.157195 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:13.260065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:13.260023 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" event={"ID":"4eec0817-1add-4a0b-af21-3cbdf992c01a","Type":"ContainerStarted","Data":"13078e19d4efc1f9a715f00e8462352815aae5b852e7acef7292a6af9ffeec0c"} Oct 31 01:51:13.260479 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:13.260236 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:13.282610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:13.282551 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" podStartSLOduration=1.7962611910000001 podStartE2EDuration="2.28253736s" totalImagesPullingTime="486.276169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:13.277986529 +0000 UTC m=+1873.658231772" watchObservedRunningTime="2026-10-31 01:51:13.28253736 +0000 UTC m=+1873.662782584" Oct 31 01:51:13.875915 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:13.875853 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99_9bca4fe6-189f-4336-a71a-2e4ce715ab32/payload-processing/0.log" Oct 31 01:51:14.157393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:14.156573 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:14.925486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:14.925360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:15.653644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:15.653600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:16.477457 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:16.477307 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:17.285557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:17.285521 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:17.940707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:17.940674 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:18.758861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:18.758828 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:19.176060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.175975 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg"] Oct 31 01:51:19.177191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.176869 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" podUID="9dbf4908-d08b-4b67-9a74-c15201706548" containerName="maas-api" containerID="cri-o://b4b1cc52e443392b14f733811fa0d476c90e5947caae7d7131ffb1087803577a" gracePeriod=30 Oct 31 01:51:19.261487 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.261458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:19.273333 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.273313 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:51:19.318931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.318891 5090 generic.go:356] "Generic (PLEG): container finished" podID="9dbf4908-d08b-4b67-9a74-c15201706548" containerID="b4b1cc52e443392b14f733811fa0d476c90e5947caae7d7131ffb1087803577a" exitCode=0 Oct 31 01:51:19.319101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.319081 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" event={"ID":"9dbf4908-d08b-4b67-9a74-c15201706548","Type":"ContainerDied","Data":"b4b1cc52e443392b14f733811fa0d476c90e5947caae7d7131ffb1087803577a"} Oct 31 01:51:19.405233 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.405204 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:19.552745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.544321 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:51:19.569738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.560663 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:19.646755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.646249 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-maas-api-tls\") pod \"9dbf4908-d08b-4b67-9a74-c15201706548\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " Oct 31 01:51:19.646755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.646315 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-metrics-certs\") pod \"9dbf4908-d08b-4b67-9a74-c15201706548\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " Oct 31 01:51:19.646755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.646520 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dbf4908-d08b-4b67-9a74-c15201706548-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbf4908-d08b-4b67-9a74-c15201706548-openshift-service-ca\") pod \"9dbf4908-d08b-4b67-9a74-c15201706548\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " Oct 31 01:51:19.646755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.646597 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dbf4908-d08b-4b67-9a74-c15201706548-kube-api-access-mhr2r\" (UniqueName: \"kubernetes.io/projected/9dbf4908-d08b-4b67-9a74-c15201706548-kube-api-access-mhr2r\") pod \"9dbf4908-d08b-4b67-9a74-c15201706548\" (UID: \"9dbf4908-d08b-4b67-9a74-c15201706548\") " Oct 31 01:51:19.666836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.655967 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dbf4908-d08b-4b67-9a74-c15201706548-openshift-service-ca" pod "9dbf4908-d08b-4b67-9a74-c15201706548" (UID: "9dbf4908-d08b-4b67-9a74-c15201706548"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:51:19.666836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.657337 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-metrics-certs" pod "9dbf4908-d08b-4b67-9a74-c15201706548" (UID: "9dbf4908-d08b-4b67-9a74-c15201706548"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:19.666836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.661225 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-maas-api-tls" pod "9dbf4908-d08b-4b67-9a74-c15201706548" (UID: "9dbf4908-d08b-4b67-9a74-c15201706548"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:19.666836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.662110 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dbf4908-d08b-4b67-9a74-c15201706548-kube-api-access-mhr2r" pod "9dbf4908-d08b-4b67-9a74-c15201706548" (UID: "9dbf4908-d08b-4b67-9a74-c15201706548"). InnerVolumeSpecName "kube-api-access-mhr2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:51:19.716616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.716517 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:19.783155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.783116 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dbf4908-d08b-4b67-9a74-c15201706548-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:19.783380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.783155 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhr2r\" (UniqueName: \"kubernetes.io/projected/9dbf4908-d08b-4b67-9a74-c15201706548-kube-api-access-mhr2r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:19.783380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.783192 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:19.783380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:19.783206 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9dbf4908-d08b-4b67-9a74-c15201706548-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:20.159259 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:20.159178 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:20.327371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:20.327340 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" Oct 31 01:51:20.327884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:20.327385 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg" event={"ID":"9dbf4908-d08b-4b67-9a74-c15201706548","Type":"ContainerDied","Data":"ca898f7c6206a1d7c5d9d16a1c50246c83e9a88fb460e6e1dcee0abac579908c"} Oct 31 01:51:20.327884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:20.327452 5090 scope.go:122] "RemoveContainer" containerID="b4b1cc52e443392b14f733811fa0d476c90e5947caae7d7131ffb1087803577a" Oct 31 01:51:20.345310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:20.345286 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg"] Oct 31 01:51:20.350666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:20.350639 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e52ba1-5fc66d998-xnwdg"] Oct 31 01:51:20.755037 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:20.755006 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:21.373293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:21.365630 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:21.747599 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:21.747569 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dbf4908-d08b-4b67-9a74-c15201706548" path="/var/lib/kubelet/pods/9dbf4908-d08b-4b67-9a74-c15201706548/volumes" Oct 31 01:51:22.047827 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:22.047140 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:22.799744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:22.798087 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:23.491481 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:23.491432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:24.328660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:24.323490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:25.315968 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:25.315935 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:26.064644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.064609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:26.528618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.528582 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz"] Oct 31 01:51:26.529168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.529137 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" podUID="4223400f-40ee-4acc-b758-cb23412a7724" containerName="maas-api" containerID="cri-o://1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82" gracePeriod=30 Oct 31 01:51:26.592940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.592903 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:26.730971 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.730937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:26.853492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.853462 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:51:26.879673 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.879642 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:26.903365 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.903336 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-metrics-certs\") pod \"4223400f-40ee-4acc-b758-cb23412a7724\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " Oct 31 01:51:26.903556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.903458 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4223400f-40ee-4acc-b758-cb23412a7724-kube-api-access-chqd6\" (UniqueName: \"kubernetes.io/projected/4223400f-40ee-4acc-b758-cb23412a7724-kube-api-access-chqd6\") pod \"4223400f-40ee-4acc-b758-cb23412a7724\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " Oct 31 01:51:26.903556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.903506 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4223400f-40ee-4acc-b758-cb23412a7724-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4223400f-40ee-4acc-b758-cb23412a7724-openshift-service-ca\") pod \"4223400f-40ee-4acc-b758-cb23412a7724\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " Oct 31 01:51:26.903675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.903591 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-maas-api-tls\") pod \"4223400f-40ee-4acc-b758-cb23412a7724\" (UID: \"4223400f-40ee-4acc-b758-cb23412a7724\") " Oct 31 01:51:26.905456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.905410 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4223400f-40ee-4acc-b758-cb23412a7724-openshift-service-ca" pod "4223400f-40ee-4acc-b758-cb23412a7724" (UID: "4223400f-40ee-4acc-b758-cb23412a7724"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:51:26.910707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.910576 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4223400f-40ee-4acc-b758-cb23412a7724-kube-api-access-chqd6" pod "4223400f-40ee-4acc-b758-cb23412a7724" (UID: "4223400f-40ee-4acc-b758-cb23412a7724"). InnerVolumeSpecName "kube-api-access-chqd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:51:26.910707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.910669 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-metrics-certs" pod "4223400f-40ee-4acc-b758-cb23412a7724" (UID: "4223400f-40ee-4acc-b758-cb23412a7724"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:26.910912 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:26.910785 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-maas-api-tls" pod "4223400f-40ee-4acc-b758-cb23412a7724" (UID: "4223400f-40ee-4acc-b758-cb23412a7724"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:27.017664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.014930 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:27.017664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.014971 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chqd6\" (UniqueName: \"kubernetes.io/projected/4223400f-40ee-4acc-b758-cb23412a7724-kube-api-access-chqd6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:27.017664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.014990 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4223400f-40ee-4acc-b758-cb23412a7724-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:27.017664 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.015006 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4223400f-40ee-4acc-b758-cb23412a7724-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:27.064176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.052182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:27.199792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.199256 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:27.339245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.339215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:27.387134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.387105 5090 generic.go:356] "Generic (PLEG): container finished" podID="4223400f-40ee-4acc-b758-cb23412a7724" containerID="1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82" exitCode=0 Oct 31 01:51:27.387318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.387147 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" event={"ID":"4223400f-40ee-4acc-b758-cb23412a7724","Type":"ContainerDied","Data":"1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82"} Oct 31 01:51:27.387318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.387154 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" Oct 31 01:51:27.387318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.387190 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz" event={"ID":"4223400f-40ee-4acc-b758-cb23412a7724","Type":"ContainerDied","Data":"1ea49eef558d1bc5dfdb2a2e0b45229ffd3995208ae19ab3542d8eaf1a1426e8"} Oct 31 01:51:27.387318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.387213 5090 scope.go:122] "RemoveContainer" containerID="1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82" Oct 31 01:51:27.398806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.398709 5090 scope.go:122] "RemoveContainer" containerID="1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82" Oct 31 01:51:27.399175 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:51:27.399149 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82\": container with ID starting with 1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82 not found: ID does not exist" containerID="1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82" Oct 31 01:51:27.399274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.399200 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82"} err="failed to get container status \"1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82\": rpc error: code = NotFound desc = could not find container \"1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82\": container with ID starting with 1015b0a9caf4069506642115ae786122b8dff627e38c59cb8a1ce50453018c82 not found: ID does not exist" Oct 31 01:51:27.414040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.414000 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz"] Oct 31 01:51:27.417504 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.417481 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-90c4d1-656fc865d-q84mz"] Oct 31 01:51:27.485399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.485304 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:27.751540 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.751505 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4223400f-40ee-4acc-b758-cb23412a7724" path="/var/lib/kubelet/pods/4223400f-40ee-4acc-b758-cb23412a7724/volumes" Oct 31 01:51:27.945358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:27.945325 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:28.132069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:28.131986 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:28.269927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:28.269895 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:28.706485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:28.706444 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:28.927026 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:28.925119 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:29.079136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:29.078850 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:29.549050 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:29.549014 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:30.279607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.277646 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz"] Oct 31 01:51:30.279607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.278637 5090 cpu_manager.go:414] "containerMap: removing container" podUID="4223400f-40ee-4acc-b758-cb23412a7724" containerName="maas-api" Oct 31 01:51:30.279607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.278658 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4223400f-40ee-4acc-b758-cb23412a7724" containerName="maas-api" Oct 31 01:51:30.279607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.278769 5090 cpu_manager.go:414] "containerMap: removing container" podUID="9dbf4908-d08b-4b67-9a74-c15201706548" containerName="maas-api" Oct 31 01:51:30.279607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.278778 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dbf4908-d08b-4b67-9a74-c15201706548" containerName="maas-api" Oct 31 01:51:30.279607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.278898 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dbf4908-d08b-4b67-9a74-c15201706548" containerName="maas-api" Oct 31 01:51:30.279607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.278918 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="4223400f-40ee-4acc-b758-cb23412a7724" containerName="maas-api" Oct 31 01:51:30.285329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.284522 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.306784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.302892 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a779b3\"/\"openshift-service-ca.crt\"" Oct 31 01:51:30.306784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.302984 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a779b3\"/\"praxis-route-a779b3-kserve-self-signed-certs\"" Oct 31 01:51:30.306784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.303120 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-a779b3\"/\"default-dockercfg-mpjk4\"" Oct 31 01:51:30.306784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.303142 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-a779b3\"/\"kube-root-ca.crt\"" Oct 31 01:51:30.306784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.305880 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz"] Oct 31 01:51:30.318189 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.318165 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:30.412012 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.411960 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-dshm\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.412221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.412136 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-home\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.412221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.412174 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-model-cache\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.412327 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.412305 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zgzn\" (UniqueName: \"kubernetes.io/projected/826a4006-be1b-435c-b8c8-f65b5b6c548c-kube-api-access-4zgzn\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.412418 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.412401 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-tmp-dir\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.412533 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.412514 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826a4006-be1b-435c-b8c8-f65b5b6c548c-tls-certs\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.463683 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.463648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:30.516678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.513688 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-dshm\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.516678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.513841 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-home\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.516678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.513880 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-model-cache\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.516678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.513928 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zgzn\" (UniqueName: \"kubernetes.io/projected/826a4006-be1b-435c-b8c8-f65b5b6c548c-kube-api-access-4zgzn\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.516678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.514017 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-tmp-dir\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.516678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.514101 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826a4006-be1b-435c-b8c8-f65b5b6c548c-tls-certs\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.516678 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.515159 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-model-cache\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.527736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.519087 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-home\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.541780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.538442 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-tmp-dir\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.541780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.538861 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-dshm\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.541780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.539468 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826a4006-be1b-435c-b8c8-f65b5b6c548c-tls-certs\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.572824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.566853 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zgzn\" (UniqueName: \"kubernetes.io/projected/826a4006-be1b-435c-b8c8-f65b5b6c548c-kube-api-access-4zgzn\") pod \"praxis-route-a779b3-kserve-5588676559-d2fsz\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:30.631752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:30.629015 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:31.046878 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:31.046848 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz"] Oct 31 01:51:31.047941 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:51:31.047912 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod826a4006_be1b_435c_b8c8_f65b5b6c548c.slice/crio-3ba898135f5bdb7b46ca572b98666e21c58899f874336b30b13d8fb3b56796d4 WatchSource:0}: Error finding container 3ba898135f5bdb7b46ca572b98666e21c58899f874336b30b13d8fb3b56796d4: Status 404 returned error can't find the container with id 3ba898135f5bdb7b46ca572b98666e21c58899f874336b30b13d8fb3b56796d4 Oct 31 01:51:31.328766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:31.328710 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:31.421937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:31.421156 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" event={"ID":"826a4006-be1b-435c-b8c8-f65b5b6c548c","Type":"ContainerStarted","Data":"afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e"} Oct 31 01:51:31.421937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:31.421223 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" event={"ID":"826a4006-be1b-435c-b8c8-f65b5b6c548c","Type":"ContainerStarted","Data":"3ba898135f5bdb7b46ca572b98666e21c58899f874336b30b13d8fb3b56796d4"} Oct 31 01:51:31.443267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:31.443219 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" podStartSLOduration=1.44320573 podStartE2EDuration="1.44320573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:31.439771211 +0000 UTC m=+1891.820016464" watchObservedRunningTime="2026-10-31 01:51:31.44320573 +0000 UTC m=+1891.823450954" Oct 31 01:51:31.633913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:31.633876 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:32.398827 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:32.398780 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:32.703793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:32.683504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:33.466464 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:33.464922 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:33.604017 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:33.603961 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:34.393741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:34.390608 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:34.613612 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:34.613560 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:34.751466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:34.751440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:34.789762 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:51:34.789700 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:51:34.987650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:34.987430 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:35.478347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:35.476972 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:36.157251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:36.157211 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:36.598800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:36.598769 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:37.118641 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:37.118610 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:37.273380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:37.273349 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:37.447831 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:37.445000 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:37.849626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:37.849592 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:38.560944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:38.560914 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:39.255884 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:39.255842 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:40.072199 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:40.072165 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:40.602371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:40.602334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:40.629686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:40.629653 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:40.642890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:40.642869 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:40.748990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:40.748959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:41.036205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:41.036169 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:41.180083 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:41.180050 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:41.336925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:41.336837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:41.555962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:41.555934 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:51:42.068738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:42.068690 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:42.626579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:42.626550 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:42.800663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:42.800637 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:42.946001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:42.945924 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:43.093506 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:43.093474 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:43.636480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:43.636405 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:44.420276 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:44.420244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:45.199061 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:45.198983 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:45.679271 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:45.679236 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw"] Oct 31 01:51:45.679784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:45.679540 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" podUID="e6a699d3-68ab-434a-8c40-de3b0afde7b8" containerName="maas-api" containerID="cri-o://c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d" gracePeriod=30 Oct 31 01:51:45.757306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:45.757276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:45.915503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:45.914783 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:46.052755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.036043 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:51:46.061585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.061508 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:46.132243 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.132207 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a699d3-68ab-434a-8c40-de3b0afde7b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a699d3-68ab-434a-8c40-de3b0afde7b8-openshift-service-ca\") pod \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " Oct 31 01:51:46.132911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.132888 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a699d3-68ab-434a-8c40-de3b0afde7b8-kube-api-access-vm8jt\" (UniqueName: \"kubernetes.io/projected/e6a699d3-68ab-434a-8c40-de3b0afde7b8-kube-api-access-vm8jt\") pod \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " Oct 31 01:51:46.133415 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.133396 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-metrics-certs\") pod \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " Oct 31 01:51:46.134036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.134017 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-maas-api-tls\") pod \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\" (UID: \"e6a699d3-68ab-434a-8c40-de3b0afde7b8\") " Oct 31 01:51:46.140744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.132800 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a699d3-68ab-434a-8c40-de3b0afde7b8-openshift-service-ca" pod "e6a699d3-68ab-434a-8c40-de3b0afde7b8" (UID: "e6a699d3-68ab-434a-8c40-de3b0afde7b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:51:46.141411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.141372 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-metrics-certs" pod "e6a699d3-68ab-434a-8c40-de3b0afde7b8" (UID: "e6a699d3-68ab-434a-8c40-de3b0afde7b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:46.141707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.141683 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a699d3-68ab-434a-8c40-de3b0afde7b8-kube-api-access-vm8jt" pod "e6a699d3-68ab-434a-8c40-de3b0afde7b8" (UID: "e6a699d3-68ab-434a-8c40-de3b0afde7b8"). InnerVolumeSpecName "kube-api-access-vm8jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:51:46.154568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.154510 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-maas-api-tls" pod "e6a699d3-68ab-434a-8c40-de3b0afde7b8" (UID: "e6a699d3-68ab-434a-8c40-de3b0afde7b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:46.240807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.237213 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6a699d3-68ab-434a-8c40-de3b0afde7b8-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:46.240807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.237253 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8jt\" (UniqueName: \"kubernetes.io/projected/e6a699d3-68ab-434a-8c40-de3b0afde7b8-kube-api-access-vm8jt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:46.240807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.237271 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:46.240807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.237286 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6a699d3-68ab-434a-8c40-de3b0afde7b8-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:46.315429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.315396 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:46.550846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.550811 5090 generic.go:356] "Generic (PLEG): container finished" podID="e6a699d3-68ab-434a-8c40-de3b0afde7b8" containerID="c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d" exitCode=0 Oct 31 01:51:46.551063 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.550928 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" event={"ID":"e6a699d3-68ab-434a-8c40-de3b0afde7b8","Type":"ContainerDied","Data":"c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d"} Oct 31 01:51:46.551063 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.550983 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" event={"ID":"e6a699d3-68ab-434a-8c40-de3b0afde7b8","Type":"ContainerDied","Data":"0fe65e952849bdb0044652fb5b56baff5604b61002b49449e8b4ea35f8852d17"} Oct 31 01:51:46.551063 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.551013 5090 scope.go:122] "RemoveContainer" containerID="c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d" Oct 31 01:51:46.551204 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.551146 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw" Oct 31 01:51:46.562669 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.562650 5090 scope.go:122] "RemoveContainer" containerID="c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d" Oct 31 01:51:46.562963 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:51:46.562941 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d\": container with ID starting with c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d not found: ID does not exist" containerID="c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d" Oct 31 01:51:46.563012 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.562975 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d"} err="failed to get container status \"c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d\": rpc error: code = NotFound desc = could not find container \"c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d\": container with ID starting with c3591e3224be14ed49e8893f04342335750514485e1ae620100987194fe9b94d not found: ID does not exist" Oct 31 01:51:46.570977 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.570954 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:46.577585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.577561 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw"] Oct 31 01:51:46.582617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.582592 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-611693-5659748b96-4dqlw"] Oct 31 01:51:46.824988 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.824909 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j"] Oct 31 01:51:46.825489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.825176 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" podUID="d4c2635d-a69a-462e-97eb-04a0ced1a872" containerName="maas-api" containerID="cri-o://7ac0d7e0b2003cff7011f57d3fca7c25d7fe5c593e06ec5b7fb582464035f49b" gracePeriod=30 Oct 31 01:51:46.943167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.943132 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:46.962049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.961257 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99"] Oct 31 01:51:46.962049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.961531 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" podUID="9bca4fe6-189f-4336-a71a-2e4ce715ab32" containerName="payload-processing" containerID="cri-o://0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a" gracePeriod=30 Oct 31 01:51:46.966057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.965793 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4"] Oct 31 01:51:46.966057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:46.966014 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" podUID="e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" containerName="payload-pre-processing" containerID="cri-o://f696cf3ea76be5dbd80e3ff4213781354b94cabab15f4a33fa0613c8aca1039f" gracePeriod=30 Oct 31 01:51:47.091351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.091318 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:47.354800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.351741 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:47.495496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.495463 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:47.635365 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.635275 5090 generic.go:356] "Generic (PLEG): container finished" podID="d4c2635d-a69a-462e-97eb-04a0ced1a872" containerID="7ac0d7e0b2003cff7011f57d3fca7c25d7fe5c593e06ec5b7fb582464035f49b" exitCode=0 Oct 31 01:51:47.635540 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.635448 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" event={"ID":"d4c2635d-a69a-462e-97eb-04a0ced1a872","Type":"ContainerDied","Data":"7ac0d7e0b2003cff7011f57d3fca7c25d7fe5c593e06ec5b7fb582464035f49b"} Oct 31 01:51:47.636103 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.636079 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:51:47.640258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.640206 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:47.643794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.642135 5090 generic.go:356] "Generic (PLEG): container finished" podID="9bca4fe6-189f-4336-a71a-2e4ce715ab32" containerID="0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a" exitCode=0 Oct 31 01:51:47.643794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.642373 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" event={"ID":"9bca4fe6-189f-4336-a71a-2e4ce715ab32","Type":"ContainerDied","Data":"0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a"} Oct 31 01:51:47.643794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.642427 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" event={"ID":"9bca4fe6-189f-4336-a71a-2e4ce715ab32","Type":"ContainerDied","Data":"e30c4bbbd00b95bf2c3c02bc17c4f43eb04c758b4294c50303607d9ddddf628e"} Oct 31 01:51:47.643794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.642465 5090 scope.go:122] "RemoveContainer" containerID="0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a" Oct 31 01:51:47.648906 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.648874 5090 generic.go:356] "Generic (PLEG): container finished" podID="e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" containerID="f696cf3ea76be5dbd80e3ff4213781354b94cabab15f4a33fa0613c8aca1039f" exitCode=0 Oct 31 01:51:47.649040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.648960 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" event={"ID":"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c","Type":"ContainerDied","Data":"f696cf3ea76be5dbd80e3ff4213781354b94cabab15f4a33fa0613c8aca1039f"} Oct 31 01:51:47.676573 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.676542 5090 scope.go:122] "RemoveContainer" containerID="0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a" Oct 31 01:51:47.678180 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:51:47.678146 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a\": container with ID starting with 0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a not found: ID does not exist" containerID="0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a" Oct 31 01:51:47.678360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.678331 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a"} err="failed to get container status \"0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a\": rpc error: code = NotFound desc = could not find container \"0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a\": container with ID starting with 0f0c5bfd333a14f32fbf9b6b6db2a49770629311677c251f0b8419c3af9e739a not found: ID does not exist" Oct 31 01:51:47.692361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.692333 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:51:47.749437 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.749405 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a699d3-68ab-434a-8c40-de3b0afde7b8" path="/var/lib/kubelet/pods/e6a699d3-68ab-434a-8c40-de3b0afde7b8/volumes" Oct 31 01:51:47.761189 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.761165 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:51:47.796492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.796458 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-kube-api-access-rtvcx\" (UniqueName: \"kubernetes.io/projected/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-kube-api-access-rtvcx\") pod \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " Oct 31 01:51:47.797349 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.797321 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bca4fe6-189f-4336-a71a-2e4ce715ab32-kube-api-access-cq9qn\" (UniqueName: \"kubernetes.io/projected/9bca4fe6-189f-4336-a71a-2e4ce715ab32-kube-api-access-cq9qn\") pod \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " Oct 31 01:51:47.797610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.797555 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bca4fe6-189f-4336-a71a-2e4ce715ab32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bca4fe6-189f-4336-a71a-2e4ce715ab32-plugins-config-volume\") pod \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\" (UID: \"9bca4fe6-189f-4336-a71a-2e4ce715ab32\") " Oct 31 01:51:47.798420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.798274 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bca4fe6-189f-4336-a71a-2e4ce715ab32-plugins-config-volume" pod "9bca4fe6-189f-4336-a71a-2e4ce715ab32" (UID: "9bca4fe6-189f-4336-a71a-2e4ce715ab32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:51:47.798544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.798525 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-plugins-config-volume\") pod \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\" (UID: \"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c\") " Oct 31 01:51:47.798820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.798793 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-plugins-config-volume" pod "e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" (UID: "e0e15e12-049d-4c0d-a3e7-bd56dda3e40c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:51:47.799510 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.799486 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bca4fe6-189f-4336-a71a-2e4ce715ab32-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:47.799594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.799515 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:47.799774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.799750 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bca4fe6-189f-4336-a71a-2e4ce715ab32-kube-api-access-cq9qn" pod "9bca4fe6-189f-4336-a71a-2e4ce715ab32" (UID: "9bca4fe6-189f-4336-a71a-2e4ce715ab32"). InnerVolumeSpecName "kube-api-access-cq9qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:51:47.800856 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.800831 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-kube-api-access-rtvcx" pod "e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" (UID: "e0e15e12-049d-4c0d-a3e7-bd56dda3e40c"). InnerVolumeSpecName "kube-api-access-rtvcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:51:47.900840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.900763 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c2635d-a69a-462e-97eb-04a0ced1a872-kube-api-access-cxzt9\" (UniqueName: \"kubernetes.io/projected/d4c2635d-a69a-462e-97eb-04a0ced1a872-kube-api-access-cxzt9\") pod \"d4c2635d-a69a-462e-97eb-04a0ced1a872\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " Oct 31 01:51:47.900840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.900817 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-maas-api-tls\") pod \"d4c2635d-a69a-462e-97eb-04a0ced1a872\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " Oct 31 01:51:47.902113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.900930 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c2635d-a69a-462e-97eb-04a0ced1a872-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c2635d-a69a-462e-97eb-04a0ced1a872-openshift-service-ca\") pod \"d4c2635d-a69a-462e-97eb-04a0ced1a872\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " Oct 31 01:51:47.902113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.900967 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-metrics-certs\") pod \"d4c2635d-a69a-462e-97eb-04a0ced1a872\" (UID: \"d4c2635d-a69a-462e-97eb-04a0ced1a872\") " Oct 31 01:51:47.902113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.901456 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvcx\" (UniqueName: \"kubernetes.io/projected/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c-kube-api-access-rtvcx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:47.902113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.901479 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9qn\" (UniqueName: \"kubernetes.io/projected/9bca4fe6-189f-4336-a71a-2e4ce715ab32-kube-api-access-cq9qn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:47.902113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.901489 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c2635d-a69a-462e-97eb-04a0ced1a872-openshift-service-ca" pod "d4c2635d-a69a-462e-97eb-04a0ced1a872" (UID: "d4c2635d-a69a-462e-97eb-04a0ced1a872"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:51:47.912625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.909685 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-maas-api-tls" pod "d4c2635d-a69a-462e-97eb-04a0ced1a872" (UID: "d4c2635d-a69a-462e-97eb-04a0ced1a872"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:47.912625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.910091 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c2635d-a69a-462e-97eb-04a0ced1a872-kube-api-access-cxzt9" pod "d4c2635d-a69a-462e-97eb-04a0ced1a872" (UID: "d4c2635d-a69a-462e-97eb-04a0ced1a872"). InnerVolumeSpecName "kube-api-access-cxzt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:51:47.912625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:47.912082 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-metrics-certs" pod "d4c2635d-a69a-462e-97eb-04a0ced1a872" (UID: "d4c2635d-a69a-462e-97eb-04a0ced1a872"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:51:48.003166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.003140 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxzt9\" (UniqueName: \"kubernetes.io/projected/d4c2635d-a69a-462e-97eb-04a0ced1a872-kube-api-access-cxzt9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:48.003166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.003167 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:48.003323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.003177 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4c2635d-a69a-462e-97eb-04a0ced1a872-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:48.003323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.003186 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4c2635d-a69a-462e-97eb-04a0ced1a872-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:51:48.007443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.007413 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:48.456748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.456258 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:48.656666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.656641 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99" Oct 31 01:51:48.658271 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.658249 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" Oct 31 01:51:48.658404 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.658279 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4" event={"ID":"e0e15e12-049d-4c0d-a3e7-bd56dda3e40c","Type":"ContainerDied","Data":"65b453182163888bcab054763f1ce6eb4aeaed2f56be84c3908da1f47574b5d2"} Oct 31 01:51:48.658404 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.658354 5090 scope.go:122] "RemoveContainer" containerID="f696cf3ea76be5dbd80e3ff4213781354b94cabab15f4a33fa0613c8aca1039f" Oct 31 01:51:48.659881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.659828 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" event={"ID":"d4c2635d-a69a-462e-97eb-04a0ced1a872","Type":"ContainerDied","Data":"ed8827a7a484ce2023084c6bc5e19bfcde66ca57edbc2c02f05f74968e6cc814"} Oct 31 01:51:48.659972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.659886 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j" Oct 31 01:51:48.671316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.671299 5090 scope.go:122] "RemoveContainer" containerID="7ac0d7e0b2003cff7011f57d3fca7c25d7fe5c593e06ec5b7fb582464035f49b" Oct 31 01:51:48.688919 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.688894 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99"] Oct 31 01:51:48.694357 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.694338 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-03004a-5d58b9c94-bzv99"] Oct 31 01:51:48.707586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.707535 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j"] Oct 31 01:51:48.716825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.716807 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-03004a-6c755b4958-6dl8j"] Oct 31 01:51:48.726438 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.726416 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4"] Oct 31 01:51:48.738086 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:48.738062 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-03004a-f9866d4cf-fwfh4"] Oct 31 01:51:49.249910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.249867 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:49.754925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.754886 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bca4fe6-189f-4336-a71a-2e4ce715ab32" path="/var/lib/kubelet/pods/9bca4fe6-189f-4336-a71a-2e4ce715ab32/volumes" Oct 31 01:51:49.755628 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.755598 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c2635d-a69a-462e-97eb-04a0ced1a872" path="/var/lib/kubelet/pods/d4c2635d-a69a-462e-97eb-04a0ced1a872/volumes" Oct 31 01:51:49.756205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.756182 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" path="/var/lib/kubelet/pods/e0e15e12-049d-4c0d-a3e7-bd56dda3e40c/volumes" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.757691 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm"] Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.758581 5090 cpu_manager.go:414] "containerMap: removing container" podUID="d4c2635d-a69a-462e-97eb-04a0ced1a872" containerName="maas-api" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.758609 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c2635d-a69a-462e-97eb-04a0ced1a872" containerName="maas-api" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.758644 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e6a699d3-68ab-434a-8c40-de3b0afde7b8" containerName="maas-api" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.758652 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a699d3-68ab-434a-8c40-de3b0afde7b8" containerName="maas-api" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.758670 5090 cpu_manager.go:414] "containerMap: removing container" podUID="9bca4fe6-189f-4336-a71a-2e4ce715ab32" containerName="payload-processing" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.758679 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bca4fe6-189f-4336-a71a-2e4ce715ab32" containerName="payload-processing" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.759276 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" containerName="payload-pre-processing" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.759292 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" containerName="payload-pre-processing" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.760601 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c2635d-a69a-462e-97eb-04a0ced1a872" containerName="maas-api" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.760648 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a699d3-68ab-434a-8c40-de3b0afde7b8" containerName="maas-api" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.760659 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bca4fe6-189f-4336-a71a-2e4ce715ab32" containerName="payload-processing" Oct 31 01:51:49.761613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.760674 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e15e12-049d-4c0d-a3e7-bd56dda3e40c" containerName="payload-pre-processing" Oct 31 01:51:49.766684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.766664 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm"] Oct 31 01:51:49.766842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.766778 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:49.772185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.772166 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-b8d890-serving-cert\"" Oct 31 01:51:49.864685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.859094 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:49.928998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.928960 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-metrics-certs\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:49.929181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.929066 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnjw\" (UniqueName: \"kubernetes.io/projected/39670190-f363-4dc3-9cb8-2955fb969799-kube-api-access-qmnjw\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:49.929279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.929177 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:49.929279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:49.929251 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39670190-f363-4dc3-9cb8-2955fb969799-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.009048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.008970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:50.030442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.030414 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-metrics-certs\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.030596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.030446 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnjw\" (UniqueName: \"kubernetes.io/projected/39670190-f363-4dc3-9cb8-2955fb969799-kube-api-access-qmnjw\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.030596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.030472 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.030596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.030516 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39670190-f363-4dc3-9cb8-2955fb969799-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.031215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.031188 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39670190-f363-4dc3-9cb8-2955fb969799-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.033165 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.033143 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-maas-api-tls\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.033277 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.033183 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-metrics-certs\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.042223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.042199 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnjw\" (UniqueName: \"kubernetes.io/projected/39670190-f363-4dc3-9cb8-2955fb969799-kube-api-access-qmnjw\") pod \"maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.090336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.090304 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:50.164527 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.159553 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:50.458681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.458589 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm"] Oct 31 01:51:50.466887 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:51:50.466583 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39670190_f363_4dc3_9cb8_2955fb969799.slice/crio-ce43910518fd7499c8ffc2196a7cf784ac6da5fe1c7dffa0ee66f351ab43666b WatchSource:0}: Error finding container ce43910518fd7499c8ffc2196a7cf784ac6da5fe1c7dffa0ee66f351ab43666b: Status 404 returned error can't find the container with id ce43910518fd7499c8ffc2196a7cf784ac6da5fe1c7dffa0ee66f351ab43666b Oct 31 01:51:50.596828 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.596799 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:50.677307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.677270 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" event={"ID":"39670190-f363-4dc3-9cb8-2955fb969799","Type":"ContainerStarted","Data":"ce43910518fd7499c8ffc2196a7cf784ac6da5fe1c7dffa0ee66f351ab43666b"} Oct 31 01:51:50.753321 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:50.745575 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:51.374473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:51.374430 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:51.686140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:51.686055 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" event={"ID":"39670190-f363-4dc3-9cb8-2955fb969799","Type":"ContainerStarted","Data":"94710ce61c7dd5c27e8fc703da117cc616074c3932fbf68f22b45d83147af037"} Oct 31 01:51:51.686509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:51.686147 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:51.714411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:51.714359 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" podStartSLOduration=2.294756012 podStartE2EDuration="2.714344848s" totalImagesPullingTime="419.588836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:51:51.71030705 +0000 UTC m=+1912.090552292" watchObservedRunningTime="2026-10-31 01:51:51.714344848 +0000 UTC m=+1912.094590071" Oct 31 01:51:52.115787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:52.115745 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:52.601466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:52.601438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:52.736462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:52.736428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:52.909791 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:52.891055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:53.076758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:53.073137 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:53.210358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:53.210272 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:53.368379 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:53.368332 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:54.000397 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:53.995544 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:54.768088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:54.766372 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:55.543127 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:55.543094 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:56.363750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:56.360305 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:57.053739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:57.053687 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:57.697445 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:57.697417 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:51:57.787826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:57.787794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:58.578747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:58.578684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:51:59.629499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:51:59.629460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:00.570284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:00.570247 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:00.846252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:00.846215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:01.326894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:01.326860 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:01.913631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:01.913602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:02.063044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:02.063000 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:02.239655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:02.239566 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:02.477303 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:02.477267 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:02.650330 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:02.650255 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:03.171710 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:03.171670 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:03.449003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:03.448920 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:04.037802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:04.037768 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:04.191264 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:04.191191 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:04.481855 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:04.481773 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:04.639452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:04.639419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:04.778939 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:04.778867 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:04.792091 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:52:04.792059 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:52:05.202611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:05.202533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:05.436988 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:05.436947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:06.141942 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:06.141910 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:06.442432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:06.442346 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:06.581624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:06.581593 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:06.959563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:06.959509 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:07.665799 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:07.665752 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:07.849737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:07.839223 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:08.540603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:08.540574 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:09.198922 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:09.198893 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:09.819211 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:09.819177 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:10.631170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:10.631122 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:11.406995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:11.406963 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:11.550788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:11.550703 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:11.753423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:11.753387 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:12.358781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:12.358744 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:13.041685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:13.041650 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:13.863618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:13.863584 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm"] Oct 31 01:52:13.872692 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:13.872661 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:13.873374 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:13.873349 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm"] Oct 31 01:52:13.876341 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:13.876309 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-278a56-serving-cert\"" Oct 31 01:52:13.936780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:13.936752 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:14.022662 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.022610 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fw98\" (UniqueName: \"kubernetes.io/projected/99494548-74ea-4d06-8547-fb6ce0270bbc-kube-api-access-6fw98\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.022872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.022697 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-metrics-certs\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.022872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.022797 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.022983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.022908 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99494548-74ea-4d06-8547-fb6ce0270bbc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.085074 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.085037 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:14.124652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.124582 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fw98\" (UniqueName: \"kubernetes.io/projected/99494548-74ea-4d06-8547-fb6ce0270bbc-kube-api-access-6fw98\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.124652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.124628 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-metrics-certs\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.124905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.124673 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.124905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.124784 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99494548-74ea-4d06-8547-fb6ce0270bbc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.125509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.125435 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99494548-74ea-4d06-8547-fb6ce0270bbc-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.127316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.127290 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-metrics-certs\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.127411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.127362 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-maas-api-tls\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.136137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.136113 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fw98\" (UniqueName: \"kubernetes.io/projected/99494548-74ea-4d06-8547-fb6ce0270bbc-kube-api-access-6fw98\") pod \"maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.204171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.204143 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:14.216636 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.216613 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:14.519739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.519231 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm"] Oct 31 01:52:14.562908 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.554070 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:52:14.911792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.904100 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:14.951049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:14.951006 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" event={"ID":"99494548-74ea-4d06-8547-fb6ce0270bbc","Type":"ContainerStarted","Data":"2157db430ad98d86e9beb7876bb797e3ff2753046f7569a863343b200ce19126"} Oct 31 01:52:15.722960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:15.720974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:15.959171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:15.959139 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" event={"ID":"99494548-74ea-4d06-8547-fb6ce0270bbc","Type":"ContainerStarted","Data":"f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13"} Oct 31 01:52:15.959363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:15.959341 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:15.980495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:15.980400 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" podStartSLOduration=2.38532975 podStartE2EDuration="2.980387177s" totalImagesPullingTime="595.057427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:15.977642333 +0000 UTC m=+1936.357887580" watchObservedRunningTime="2026-10-31 01:52:15.980387177 +0000 UTC m=+1936.360632400" Oct 31 01:52:16.450599 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:16.443733 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:17.125619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:17.125516 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:17.857796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:17.853326 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:18.510944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:18.510893 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:19.173769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:19.173698 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:19.679933 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:19.679900 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:19.969681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:19.969585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:20.222156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:20.222002 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:20.380760 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:20.372706 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:20.937753 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:20.935653 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:21.835605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:21.835571 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:21.971442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:21.971413 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:52:22.564588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:22.564055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:23.542100 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:23.542000 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:23.709233 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:23.709181 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:23.863946 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:23.863858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:24.171415 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:24.171327 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:24.491254 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:24.491167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:24.996679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:24.996637 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:25.190739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:25.186803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:25.523191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:25.523150 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:26.438770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:26.438730 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:26.631832 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:26.631706 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:27.362740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:27.362616 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:28.133340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:28.133239 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:29.401609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:29.401572 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:30.145806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:30.136905 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:30.300450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:30.300416 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:30.931650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:30.931616 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:31.101624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:31.101591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:31.678857 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:31.678822 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:32.452792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:32.452758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:33.267801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.267765 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq"] Oct 31 01:52:33.273196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.273169 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq"] Oct 31 01:52:33.273345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.273266 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.279016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.278995 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-925bac-serving-cert\"" Oct 31 01:52:33.331624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.331596 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stp6b\" (UniqueName: \"kubernetes.io/projected/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-kube-api-access-stp6b\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.331909 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.331883 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.331959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.331926 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.332020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.332007 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-metrics-certs\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.361698 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.361604 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:33.433302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.433272 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.433302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.433308 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.433543 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.433381 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-metrics-certs\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.433543 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.433481 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stp6b\" (UniqueName: \"kubernetes.io/projected/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-kube-api-access-stp6b\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.434098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.434074 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.436117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.436094 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.436250 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.436230 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-metrics-certs\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.447089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.447063 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stp6b\" (UniqueName: \"kubernetes.io/projected/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-kube-api-access-stp6b\") pod \"maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.543886 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.543615 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:33.595142 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.595107 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:33.914054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:33.910503 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq"] Oct 31 01:52:33.914422 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:52:33.914390 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec37f641_a7d2_4bf0_b8b2_4e37ff4c6a8f.slice/crio-dd80a5ee349f65c476c316d29591b9d6e9556c4c2b3d05d5183d61e594906faa WatchSource:0}: Error finding container dd80a5ee349f65c476c316d29591b9d6e9556c4c2b3d05d5183d61e594906faa: Status 404 returned error can't find the container with id dd80a5ee349f65c476c316d29591b9d6e9556c4c2b3d05d5183d61e594906faa Oct 31 01:52:34.174537 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:34.174463 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" event={"ID":"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f","Type":"ContainerStarted","Data":"dd80a5ee349f65c476c316d29591b9d6e9556c4c2b3d05d5183d61e594906faa"} Oct 31 01:52:34.227852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:34.227816 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:34.790170 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:52:34.790123 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:52:35.057256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:35.053140 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:35.184676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:35.184642 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" event={"ID":"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f","Type":"ContainerStarted","Data":"276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2"} Oct 31 01:52:35.207402 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:35.207356 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" podStartSLOduration=1.664635905 podStartE2EDuration="2.207343633s" totalImagesPullingTime="542.707728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:35.2028806 +0000 UTC m=+1955.583125833" watchObservedRunningTime="2026-10-31 01:52:35.207343633 +0000 UTC m=+1955.587588885" Oct 31 01:52:35.862234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:35.862199 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:36.191565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:36.191491 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:36.652693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:36.652656 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:36.993240 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:36.993210 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:37.153879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.153843 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt"] Oct 31 01:52:37.159545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.159516 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.160318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.160282 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:37.179594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.179566 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt"] Oct 31 01:52:37.248098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.248007 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b8d890-openshift-default-dockercfg-fvs7g\"" Oct 31 01:52:37.295429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295384 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295436 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d161b833-9c82-4181-92b9-be2e903c278c-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295517 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d161b833-9c82-4181-92b9-be2e903c278c-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295588 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mbg\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-kube-api-access-57mbg\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295634 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295655 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295688 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295764 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.295863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.295797 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.344638 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.344601 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:37.397185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397147 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397224 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d161b833-9c82-4181-92b9-be2e903c278c-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397292 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d161b833-9c82-4181-92b9-be2e903c278c-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397362 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mbg\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-kube-api-access-57mbg\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397408 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397436 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397466 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397529 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397563 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.397873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397622 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-credential-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.398014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.397991 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-socket\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.398078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.398024 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d161b833-9c82-4181-92b9-be2e903c278c-istiod-ca-cert\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.398078 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.398059 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-data\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.398360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.398337 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-certs\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.400203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.400169 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d161b833-9c82-4181-92b9-be2e903c278c-istio-podinfo\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.400338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.400179 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-envoy\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.410651 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.410630 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-istio-token\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.410793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.410778 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mbg\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-kube-api-access-57mbg\") pod \"e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.479663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.479634 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:37.480302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.480270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:37.831176 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.831145 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt"] Oct 31 01:52:37.832214 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:52:37.832189 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd161b833_9c82_4181_92b9_be2e903c278c.slice/crio-6768f28e75e0c2c947b3ff033cffcb426eb837a31095888a09fc62efe5a08879 WatchSource:0}: Error finding container 6768f28e75e0c2c947b3ff033cffcb426eb837a31095888a09fc62efe5a08879: Status 404 returned error can't find the container with id 6768f28e75e0c2c947b3ff033cffcb426eb837a31095888a09fc62efe5a08879 Oct 31 01:52:37.835150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.835114 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:52:37.835247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.835188 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:52:37.835247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:37.835223 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:52:38.219838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.219705 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" event={"ID":"d161b833-9c82-4181-92b9-be2e903c278c","Type":"ContainerStarted","Data":"cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef"} Oct 31 01:52:38.219838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.219810 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" event={"ID":"d161b833-9c82-4181-92b9-be2e903c278c","Type":"ContainerStarted","Data":"6768f28e75e0c2c947b3ff033cffcb426eb837a31095888a09fc62efe5a08879"} Oct 31 01:52:38.257975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.248733 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podStartSLOduration=1.24869309 podStartE2EDuration="1.24869309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:52:38.248508426 +0000 UTC m=+1958.628753650" watchObservedRunningTime="2026-10-31 01:52:38.24869309 +0000 UTC m=+1958.628938315" Oct 31 01:52:38.257975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.249361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:38.480847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.480754 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:38.482108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.482082 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 01:52:38.482287 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.482141 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 01:52:38.520388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.520360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:38.665642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.665607 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:38.865568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:38.865535 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:39.002949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.002917 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:39.164807 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.145695 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:39.484958 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.484870 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 01:52:39.485570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.485537 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 01:52:39.634818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.634786 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm"] Oct 31 01:52:39.635293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.635263 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" podUID="39670190-f363-4dc3-9cb8-2955fb969799" containerName="maas-api" containerID="cri-o://94710ce61c7dd5c27e8fc703da117cc616074c3932fbf68f22b45d83147af037" gracePeriod=30 Oct 31 01:52:39.733616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.733576 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:39.913420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:39.906490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:40.096738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.072655 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:40.261833 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.259250 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:40.276201 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.276167 5090 generic.go:356] "Generic (PLEG): container finished" podID="39670190-f363-4dc3-9cb8-2955fb969799" containerID="94710ce61c7dd5c27e8fc703da117cc616074c3932fbf68f22b45d83147af037" exitCode=0 Oct 31 01:52:40.277340 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.277312 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" event={"ID":"39670190-f363-4dc3-9cb8-2955fb969799","Type":"ContainerDied","Data":"94710ce61c7dd5c27e8fc703da117cc616074c3932fbf68f22b45d83147af037"} Oct 31 01:52:40.417763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.417734 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:52:40.496759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.491225 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-metrics-certs\") pod \"39670190-f363-4dc3-9cb8-2955fb969799\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " Oct 31 01:52:40.496759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.491492 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39670190-f363-4dc3-9cb8-2955fb969799-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39670190-f363-4dc3-9cb8-2955fb969799-openshift-service-ca\") pod \"39670190-f363-4dc3-9cb8-2955fb969799\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " Oct 31 01:52:40.496759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.491536 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-maas-api-tls\") pod \"39670190-f363-4dc3-9cb8-2955fb969799\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " Oct 31 01:52:40.496759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.491575 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39670190-f363-4dc3-9cb8-2955fb969799-kube-api-access-qmnjw\" (UniqueName: \"kubernetes.io/projected/39670190-f363-4dc3-9cb8-2955fb969799-kube-api-access-qmnjw\") pod \"39670190-f363-4dc3-9cb8-2955fb969799\" (UID: \"39670190-f363-4dc3-9cb8-2955fb969799\") " Oct 31 01:52:40.500361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.500331 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39670190-f363-4dc3-9cb8-2955fb969799-openshift-service-ca" pod "39670190-f363-4dc3-9cb8-2955fb969799" (UID: "39670190-f363-4dc3-9cb8-2955fb969799"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:52:40.502234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.502208 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 01:52:40.502573 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.502421 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 01:52:40.502892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.502862 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39670190-f363-4dc3-9cb8-2955fb969799-kube-api-access-qmnjw" pod "39670190-f363-4dc3-9cb8-2955fb969799" (UID: "39670190-f363-4dc3-9cb8-2955fb969799"). InnerVolumeSpecName "kube-api-access-qmnjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:52:40.503146 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.503120 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-metrics-certs" pod "39670190-f363-4dc3-9cb8-2955fb969799" (UID: "39670190-f363-4dc3-9cb8-2955fb969799"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:52:40.503809 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.503786 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-maas-api-tls" pod "39670190-f363-4dc3-9cb8-2955fb969799" (UID: "39670190-f363-4dc3-9cb8-2955fb969799"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:52:40.543196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.543157 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:40.597780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.594887 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39670190-f363-4dc3-9cb8-2955fb969799-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:40.597780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.594932 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:40.597780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.594949 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnjw\" (UniqueName: \"kubernetes.io/projected/39670190-f363-4dc3-9cb8-2955fb969799-kube-api-access-qmnjw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:40.597780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:40.594966 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39670190-f363-4dc3-9cb8-2955fb969799-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:41.310878 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.310848 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" Oct 31 01:52:41.311084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.310842 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm" event={"ID":"39670190-f363-4dc3-9cb8-2955fb969799","Type":"ContainerDied","Data":"ce43910518fd7499c8ffc2196a7cf784ac6da5fe1c7dffa0ee66f351ab43666b"} Oct 31 01:52:41.311084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.311000 5090 scope.go:122] "RemoveContainer" containerID="94710ce61c7dd5c27e8fc703da117cc616074c3932fbf68f22b45d83147af037" Oct 31 01:52:41.334518 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.334489 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm"] Oct 31 01:52:41.341689 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.341666 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-b8d890-5f759747fc-zngnm"] Oct 31 01:52:41.413755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.413010 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:41.495742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.495489 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 01:52:41.495742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.495603 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 01:52:41.747902 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:41.747877 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39670190-f363-4dc3-9cb8-2955fb969799" path="/var/lib/kubelet/pods/39670190-f363-4dc3-9cb8-2955fb969799/volumes" Oct 31 01:52:42.216987 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:42.216912 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:52:42.498738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:42.496109 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 01:52:42.498738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:42.496188 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 01:52:42.639742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:42.637705 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:43.497937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:43.497902 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 31 01:52:43.498432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:43.498014 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 31 01:52:43.565168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:43.564216 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:44.349407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:44.349345 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:44.742330 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:44.742239 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:44.743538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:44.743519 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:44.744510 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:44.744488 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:45.199792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:45.194793 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:46.064258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:46.056226 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:46.701392 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:46.701358 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:47.297191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:47.297154 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:47.665228 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:47.665144 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:47.839744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:47.836607 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:47.964391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:47.964299 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt"] Oct 31 01:52:47.964753 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:47.964704 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" containerID="cri-o://cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef" gracePeriod=30 Oct 31 01:52:48.038443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:48.038412 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:48.234749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:48.234627 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:48.999937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:48.999893 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:49.140642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:49.140609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:49.410385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:49.410283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:49.963062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:49.963016 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:50.570384 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:50.570344 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:50.707687 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:50.707653 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:51.345685 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:51.345644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:52.152219 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:52.152186 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:52.745003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:52.744965 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:53.257910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.257854 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:53.308850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.308816 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:53.327926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.327894 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-envoy\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.327963 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d161b833-9c82-4181-92b9-be2e903c278c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d161b833-9c82-4181-92b9-be2e903c278c-istiod-ca-cert\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.327997 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-socket\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328131 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-kube-api-access-57mbg\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-kube-api-access-57mbg\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328211 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-data\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328252 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d161b833-9c82-4181-92b9-be2e903c278c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d161b833-9c82-4181-92b9-be2e903c278c-istio-podinfo\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328291 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-istio-token\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-istio-token\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328342 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328291 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-socket" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:52:53.328503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328437 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-certs\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.328503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328466 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-data" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:52:53.328503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328486 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d161b833-9c82-4181-92b9-be2e903c278c-istiod-ca-cert" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:52:53.328667 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.328572 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-credential-socket\") pod \"d161b833-9c82-4181-92b9-be2e903c278c\" (UID: \"d161b833-9c82-4181-92b9-be2e903c278c\") " Oct 31 01:52:53.329059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.329030 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-certs" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:52:53.329166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.329056 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-credential-socket" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:52:53.329465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.329442 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.329571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.329472 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d161b833-9c82-4181-92b9-be2e903c278c-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.329571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.329488 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.329571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.329504 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.329571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.329518 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.330839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.330807 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d161b833-9c82-4181-92b9-be2e903c278c-istio-podinfo" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:52:53.331812 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.331778 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-istio-token" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:52:53.332035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.332012 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-envoy" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:52:53.332982 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.332954 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-kube-api-access-57mbg" pod "d161b833-9c82-4181-92b9-be2e903c278c" (UID: "d161b833-9c82-4181-92b9-be2e903c278c"). InnerVolumeSpecName "kube-api-access-57mbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:52:53.401434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.401403 5090 generic.go:356] "Generic (PLEG): container finished" podID="d161b833-9c82-4181-92b9-be2e903c278c" containerID="cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef" exitCode=0 Oct 31 01:52:53.401577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.401457 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" Oct 31 01:52:53.401577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.401494 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" event={"ID":"d161b833-9c82-4181-92b9-be2e903c278c","Type":"ContainerDied","Data":"cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef"} Oct 31 01:52:53.401577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.401554 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt" event={"ID":"d161b833-9c82-4181-92b9-be2e903c278c","Type":"ContainerDied","Data":"6768f28e75e0c2c947b3ff033cffcb426eb837a31095888a09fc62efe5a08879"} Oct 31 01:52:53.401679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.401586 5090 scope.go:122] "RemoveContainer" containerID="cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef" Oct 31 01:52:53.412792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.412774 5090 scope.go:122] "RemoveContainer" containerID="cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef" Oct 31 01:52:53.413029 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:52:53.413012 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef\": container with ID starting with cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef not found: ID does not exist" containerID="cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef" Oct 31 01:52:53.413081 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.413039 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef"} err="failed to get container status \"cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef\": rpc error: code = NotFound desc = could not find container \"cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef\": container with ID starting with cb51114b421f2c00a60ac41d8867405a20eecbf83981d02de65545f1a06273ef not found: ID does not exist" Oct 31 01:52:53.425863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.425838 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt"] Oct 31 01:52:53.430843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.430822 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b8d890-openshift-default-7c5f6b7449-d9vlt"] Oct 31 01:52:53.430941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.430900 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d161b833-9c82-4181-92b9-be2e903c278c-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.430941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.430918 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mbg\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-kube-api-access-57mbg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.430941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.430928 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d161b833-9c82-4181-92b9-be2e903c278c-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.430941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.430936 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d161b833-9c82-4181-92b9-be2e903c278c-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:53.569571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.567948 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:53.747835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:53.747801 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d161b833-9c82-4181-92b9-be2e903c278c" path="/var/lib/kubelet/pods/d161b833-9c82-4181-92b9-be2e903c278c/volumes" Oct 31 01:52:54.040619 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.040586 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:54.205084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.205057 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:54.326123 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.326044 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57"] Oct 31 01:52:54.326894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.326859 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" podUID="4700fe72-e630-46ff-b8fd-2adfa78b9600" containerName="maas-api" containerID="cri-o://91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555" gracePeriod=30 Oct 31 01:52:54.429416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.429383 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:54.581272 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.578911 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:54.768768 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.765177 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:54.837359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.836850 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:52:54.927310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.927276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:54.998742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.996298 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-maas-api-tls\") pod \"4700fe72-e630-46ff-b8fd-2adfa78b9600\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " Oct 31 01:52:54.998742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.996470 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4700fe72-e630-46ff-b8fd-2adfa78b9600-kube-api-access-wxfdq\" (UniqueName: \"kubernetes.io/projected/4700fe72-e630-46ff-b8fd-2adfa78b9600-kube-api-access-wxfdq\") pod \"4700fe72-e630-46ff-b8fd-2adfa78b9600\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " Oct 31 01:52:54.998742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.996535 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4700fe72-e630-46ff-b8fd-2adfa78b9600-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4700fe72-e630-46ff-b8fd-2adfa78b9600-openshift-service-ca\") pod \"4700fe72-e630-46ff-b8fd-2adfa78b9600\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " Oct 31 01:52:54.998742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:54.996698 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-metrics-certs\") pod \"4700fe72-e630-46ff-b8fd-2adfa78b9600\" (UID: \"4700fe72-e630-46ff-b8fd-2adfa78b9600\") " Oct 31 01:52:55.002057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.001492 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4700fe72-e630-46ff-b8fd-2adfa78b9600-openshift-service-ca" pod "4700fe72-e630-46ff-b8fd-2adfa78b9600" (UID: "4700fe72-e630-46ff-b8fd-2adfa78b9600"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:52:55.003987 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.003365 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-metrics-certs" pod "4700fe72-e630-46ff-b8fd-2adfa78b9600" (UID: "4700fe72-e630-46ff-b8fd-2adfa78b9600"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:52:55.004563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.004472 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-maas-api-tls" pod "4700fe72-e630-46ff-b8fd-2adfa78b9600" (UID: "4700fe72-e630-46ff-b8fd-2adfa78b9600"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:52:55.005196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.005170 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4700fe72-e630-46ff-b8fd-2adfa78b9600-kube-api-access-wxfdq" pod "4700fe72-e630-46ff-b8fd-2adfa78b9600" (UID: "4700fe72-e630-46ff-b8fd-2adfa78b9600"). InnerVolumeSpecName "kube-api-access-wxfdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:52:55.067514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.067481 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:55.099020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.098940 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxfdq\" (UniqueName: \"kubernetes.io/projected/4700fe72-e630-46ff-b8fd-2adfa78b9600-kube-api-access-wxfdq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:55.099020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.098970 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4700fe72-e630-46ff-b8fd-2adfa78b9600-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:55.099020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.098981 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:55.099020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.098990 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4700fe72-e630-46ff-b8fd-2adfa78b9600-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:52:55.438591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.438504 5090 generic.go:356] "Generic (PLEG): container finished" podID="4700fe72-e630-46ff-b8fd-2adfa78b9600" containerID="91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555" exitCode=0 Oct 31 01:52:55.438591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.438551 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" Oct 31 01:52:55.439104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.438599 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" event={"ID":"4700fe72-e630-46ff-b8fd-2adfa78b9600","Type":"ContainerDied","Data":"91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555"} Oct 31 01:52:55.439104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.438663 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57" event={"ID":"4700fe72-e630-46ff-b8fd-2adfa78b9600","Type":"ContainerDied","Data":"c5e1ab1d139ceb9b8d17ac04fb33198c583e2607b0a3d8e736fef7b2050f8817"} Oct 31 01:52:55.439104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.438698 5090 scope.go:122] "RemoveContainer" containerID="91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555" Oct 31 01:52:55.449820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.449804 5090 scope.go:122] "RemoveContainer" containerID="91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555" Oct 31 01:52:55.450099 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:52:55.450080 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555\": container with ID starting with 91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555 not found: ID does not exist" containerID="91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555" Oct 31 01:52:55.450151 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.450107 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555"} err="failed to get container status \"91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555\": rpc error: code = NotFound desc = could not find container \"91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555\": container with ID starting with 91d9278ac517fbd0ee71035595ab666e0db52188e1565fb9d81831b1be660555 not found: ID does not exist" Oct 31 01:52:55.462372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.462351 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57"] Oct 31 01:52:55.470121 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.470100 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-40a891-54cfcf88cb-t5h57"] Oct 31 01:52:55.492459 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.492437 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:55.747400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.747367 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4700fe72-e630-46ff-b8fd-2adfa78b9600" path="/var/lib/kubelet/pods/4700fe72-e630-46ff-b8fd-2adfa78b9600/volumes" Oct 31 01:52:55.838391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:55.838357 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:56.631463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:56.631425 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:57.553000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:57.552968 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:58.863785 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:58.863741 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:59.003923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:59.003893 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:59.169788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:59.169660 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:52:59.697768 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:52:59.697694 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:00.255024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:00.254995 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:00.468160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:00.468127 5090 scope.go:122] "RemoveContainer" containerID="5227bb76511d104852f24ff989364e00dc9daea22c9bbed533a5287bf07ee7de" Oct 31 01:53:01.159585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:01.159540 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:02.145491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:02.145460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:02.309964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:02.309929 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:02.468597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:02.468522 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:02.677457 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:02.677428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:02.968628 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:02.968556 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:03.761679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:03.761570 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:04.271267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:04.271235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:04.469038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:04.468993 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm"] Oct 31 01:53:04.469309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:04.469279 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" podUID="99494548-74ea-4d06-8547-fb6ce0270bbc" containerName="maas-api" containerID="cri-o://f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13" gracePeriod=30 Oct 31 01:53:04.581692 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:04.581604 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:04.794866 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:04.794834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:04.820769 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:53:04.820739 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:53:04.844156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:04.844072 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:53:05.007167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.007110 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-metrics-certs\") pod \"99494548-74ea-4d06-8547-fb6ce0270bbc\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " Oct 31 01:53:05.007499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.007478 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99494548-74ea-4d06-8547-fb6ce0270bbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99494548-74ea-4d06-8547-fb6ce0270bbc-openshift-service-ca\") pod \"99494548-74ea-4d06-8547-fb6ce0270bbc\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " Oct 31 01:53:05.007668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.007652 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-maas-api-tls\") pod \"99494548-74ea-4d06-8547-fb6ce0270bbc\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " Oct 31 01:53:05.007912 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.007896 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99494548-74ea-4d06-8547-fb6ce0270bbc-kube-api-access-6fw98\" (UniqueName: \"kubernetes.io/projected/99494548-74ea-4d06-8547-fb6ce0270bbc-kube-api-access-6fw98\") pod \"99494548-74ea-4d06-8547-fb6ce0270bbc\" (UID: \"99494548-74ea-4d06-8547-fb6ce0270bbc\") " Oct 31 01:53:05.012953 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.012924 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99494548-74ea-4d06-8547-fb6ce0270bbc-openshift-service-ca" pod "99494548-74ea-4d06-8547-fb6ce0270bbc" (UID: "99494548-74ea-4d06-8547-fb6ce0270bbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:05.052305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.052257 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99494548-74ea-4d06-8547-fb6ce0270bbc-kube-api-access-6fw98" pod "99494548-74ea-4d06-8547-fb6ce0270bbc" (UID: "99494548-74ea-4d06-8547-fb6ce0270bbc"). InnerVolumeSpecName "kube-api-access-6fw98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:05.055025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.054973 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-maas-api-tls" pod "99494548-74ea-4d06-8547-fb6ce0270bbc" (UID: "99494548-74ea-4d06-8547-fb6ce0270bbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:53:05.055486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.055462 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-metrics-certs" pod "99494548-74ea-4d06-8547-fb6ce0270bbc" (UID: "99494548-74ea-4d06-8547-fb6ce0270bbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:53:05.076181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.076080 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:05.128819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.116208 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:05.128819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.116254 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99494548-74ea-4d06-8547-fb6ce0270bbc-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:05.128819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.116269 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99494548-74ea-4d06-8547-fb6ce0270bbc-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:05.128819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.116284 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fw98\" (UniqueName: \"kubernetes.io/projected/99494548-74ea-4d06-8547-fb6ce0270bbc-kube-api-access-6fw98\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:05.256658 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.256621 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:05.304841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.304803 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6"] Oct 31 01:53:05.305954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.305880 5090 cpu_manager.go:414] "containerMap: removing container" podUID="4700fe72-e630-46ff-b8fd-2adfa78b9600" containerName="maas-api" Oct 31 01:53:05.305954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.305907 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4700fe72-e630-46ff-b8fd-2adfa78b9600" containerName="maas-api" Oct 31 01:53:05.305954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.305935 5090 cpu_manager.go:414] "containerMap: removing container" podUID="39670190-f363-4dc3-9cb8-2955fb969799" containerName="maas-api" Oct 31 01:53:05.305954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.305943 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39670190-f363-4dc3-9cb8-2955fb969799" containerName="maas-api" Oct 31 01:53:05.306262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.305966 5090 cpu_manager.go:414] "containerMap: removing container" podUID="99494548-74ea-4d06-8547-fb6ce0270bbc" containerName="maas-api" Oct 31 01:53:05.306262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.305976 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99494548-74ea-4d06-8547-fb6ce0270bbc" containerName="maas-api" Oct 31 01:53:05.306262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.306021 5090 cpu_manager.go:414] "containerMap: removing container" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" Oct 31 01:53:05.306262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.306037 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" Oct 31 01:53:05.306452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.306399 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="d161b833-9c82-4181-92b9-be2e903c278c" containerName="istio-proxy" Oct 31 01:53:05.306452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.306427 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="4700fe72-e630-46ff-b8fd-2adfa78b9600" containerName="maas-api" Oct 31 01:53:05.306452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.306443 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="39670190-f363-4dc3-9cb8-2955fb969799" containerName="maas-api" Oct 31 01:53:05.306590 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.306458 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="99494548-74ea-4d06-8547-fb6ce0270bbc" containerName="maas-api" Oct 31 01:53:05.315839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.315813 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.322244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.322217 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6"] Oct 31 01:53:05.401411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.401341 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-925bac-openshift-default-dockercfg-tr4kp\"" Oct 31 01:53:05.424533 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.424493 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.424795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.424624 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.424795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.424706 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.424980 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.424872 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.424980 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.424934 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.424980 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.424969 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.425077 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.425045 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.425181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.425164 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7np\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-kube-api-access-nd7np\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.425265 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.425251 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.506529 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.506475 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:05.526577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526542 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.526820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526590 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.526820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526614 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.526820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526668 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.526820 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526804 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7np\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-kube-api-access-nd7np\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.527035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526878 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.527035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526909 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.527035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.526954 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.527035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.527023 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.527197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.527119 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-data\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.527197 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.527143 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-credential-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.527984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.527955 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istiod-ca-cert\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.528273 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.528245 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-certs\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.528381 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.528278 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-socket\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.529910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.529885 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-podinfo\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.532325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.532301 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-envoy\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.538287 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.538257 5090 generic.go:356] "Generic (PLEG): container finished" podID="99494548-74ea-4d06-8547-fb6ce0270bbc" containerID="f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13" exitCode=0 Oct 31 01:53:05.538515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.538456 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" event={"ID":"99494548-74ea-4d06-8547-fb6ce0270bbc","Type":"ContainerDied","Data":"f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13"} Oct 31 01:53:05.538581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.538558 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" event={"ID":"99494548-74ea-4d06-8547-fb6ce0270bbc","Type":"ContainerDied","Data":"2157db430ad98d86e9beb7876bb797e3ff2753046f7569a863343b200ce19126"} Oct 31 01:53:05.538635 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.538593 5090 scope.go:122] "RemoveContainer" containerID="f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13" Oct 31 01:53:05.538809 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.538789 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm" Oct 31 01:53:05.539565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.539541 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7np\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-kube-api-access-nd7np\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.539709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.539692 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-token\") pod \"e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.562405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.562384 5090 scope.go:122] "RemoveContainer" containerID="f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13" Oct 31 01:53:05.562817 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:53:05.562788 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13\": container with ID starting with f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13 not found: ID does not exist" containerID="f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13" Oct 31 01:53:05.562912 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.562825 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13"} err="failed to get container status \"f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13\": rpc error: code = NotFound desc = could not find container \"f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13\": container with ID starting with f41f1dd8d5ef324ced21e3f15dd4670e85aa4a3af2497be9256ac9af07acea13 not found: ID does not exist" Oct 31 01:53:05.575316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.575294 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm"] Oct 31 01:53:05.585455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.585424 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-278a56-6675fb7bb6-mxttm"] Oct 31 01:53:05.655431 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.655350 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:05.756458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.756427 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99494548-74ea-4d06-8547-fb6ce0270bbc" path="/var/lib/kubelet/pods/99494548-74ea-4d06-8547-fb6ce0270bbc/volumes" Oct 31 01:53:05.948407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.948375 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6"] Oct 31 01:53:05.949781 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:53:05.949755 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151f3d99_9efa_46a4_ac2a_7dd284ed2cea.slice/crio-3e2c66cd0009eb641396971513d64ee884e4c3b5308931e3f0adc130772c1037 WatchSource:0}: Error finding container 3e2c66cd0009eb641396971513d64ee884e4c3b5308931e3f0adc130772c1037: Status 404 returned error can't find the container with id 3e2c66cd0009eb641396971513d64ee884e4c3b5308931e3f0adc130772c1037 Oct 31 01:53:05.952303 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.952270 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:53:05.952388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.952336 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:53:05.952422 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:05.952388 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:53:06.227792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:06.224226 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:06.547243 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:06.547195 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" event={"ID":"151f3d99-9efa-46a4-ac2a-7dd284ed2cea","Type":"ContainerStarted","Data":"56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc"} Oct 31 01:53:06.547437 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:06.547261 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" event={"ID":"151f3d99-9efa-46a4-ac2a-7dd284ed2cea","Type":"ContainerStarted","Data":"3e2c66cd0009eb641396971513d64ee884e4c3b5308931e3f0adc130772c1037"} Oct 31 01:53:06.570086 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:06.570016 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podStartSLOduration=1.569997117 podStartE2EDuration="1.569997117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:06.568883188 +0000 UTC m=+1986.949128435" watchObservedRunningTime="2026-10-31 01:53:06.569997117 +0000 UTC m=+1986.950242341" Oct 31 01:53:06.656401 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:06.656368 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:07.136170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:07.136137 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:07.658642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:07.658611 5090 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 01:53:07.658833 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:07.658671 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 01:53:08.522451 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:08.521304 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:08.656584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:08.656550 5090 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 01:53:08.656777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:08.656608 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 01:53:08.988397 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:08.988278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:09.180102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:09.180068 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:09.579698 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:09.579612 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:09.656362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:09.656323 5090 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 01:53:09.656671 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:09.656634 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 01:53:10.302984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:10.302951 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:10.558801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:10.546095 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:10.670170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:10.670132 5090 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 01:53:10.670788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:10.670754 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 01:53:10.934305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:10.934209 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:11.229438 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.229365 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:11.368294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.368260 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:11.656361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.656327 5090 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 01:53:11.656541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.656398 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 01:53:11.762097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.762066 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74"] Oct 31 01:53:11.767486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.767460 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.804607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.804505 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-a779b3-openshift-default-dockercfg-nkdjw\"" Oct 31 01:53:11.810738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.809882 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74"] Oct 31 01:53:11.924613 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.924436 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.926524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.926497 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.926867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.926836 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.927149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.927127 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deddf906-2736-4ef8-8c29-33e2a81bc256-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.927391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.927360 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.927649 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.927622 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5796\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-kube-api-access-n5796\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.931758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.930256 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:11.931758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.930444 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.931758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.930529 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:11.931758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:11.930581 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032036 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deddf906-2736-4ef8-8c29-33e2a81bc256-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032118 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032235 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5796\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-kube-api-access-n5796\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032348 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032479 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032431 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032537 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032487 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032630 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032590 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.032892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032872 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.033081 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.033063 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.033320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032902 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deddf906-2736-4ef8-8c29-33e2a81bc256-istiod-ca-cert\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.033513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.033490 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-credential-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.033677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.032902 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-socket\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.033891 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.033870 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-data\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.033961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.033944 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-certs\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.035552 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.035527 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-envoy\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.036899 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.036874 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-podinfo\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.046067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.046045 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-token\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.046846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.046823 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5796\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-kube-api-access-n5796\") pod \"e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.114048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.114000 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:12.459808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.459397 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74"] Oct 31 01:53:12.461044 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:53:12.461013 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeddf906_2736_4ef8_8c29_33e2a81bc256.slice/crio-6452e073ad14e833ead230e783c83a287babc45f73bf172ba4ec7d3d31eb4156 WatchSource:0}: Error finding container 6452e073ad14e833ead230e783c83a287babc45f73bf172ba4ec7d3d31eb4156: Status 404 returned error can't find the container with id 6452e073ad14e833ead230e783c83a287babc45f73bf172ba4ec7d3d31eb4156 Oct 31 01:53:12.463660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.463617 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:53:12.463781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.463680 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:53:12.463781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.463710 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:53:12.656317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.656274 5090 patch_prober.go:29] interesting pod/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 31 01:53:12.656520 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.656330 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 31 01:53:12.748393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.748357 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" event={"ID":"deddf906-2736-4ef8-8c29-33e2a81bc256","Type":"ContainerStarted","Data":"f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378"} Oct 31 01:53:12.748579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.748414 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" event={"ID":"deddf906-2736-4ef8-8c29-33e2a81bc256","Type":"ContainerStarted","Data":"6452e073ad14e833ead230e783c83a287babc45f73bf172ba4ec7d3d31eb4156"} Oct 31 01:53:12.776741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.773252 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podStartSLOduration=1.773235239 podStartE2EDuration="1.773235239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:12.770376973 +0000 UTC m=+1993.150622220" watchObservedRunningTime="2026-10-31 01:53:12.773235239 +0000 UTC m=+1993.153480461" Oct 31 01:53:12.874221 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:12.874191 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:13.024309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.024113 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:13.115089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.115046 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:13.116830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.116803 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 01:53:13.116975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.116850 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 01:53:13.250285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.250248 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:13.701742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.701038 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:13.763757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.763275 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:13.764605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.764510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:13.825049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:13.825012 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:53:14.125786 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:14.123745 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 01:53:14.125786 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:14.123808 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 01:53:14.125786 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:14.124691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:15.139739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:15.135221 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 01:53:15.139739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:15.135302 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 01:53:15.938890 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:15.938838 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:16.115170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:16.115135 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 01:53:16.115462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:16.115433 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 01:53:17.115200 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:17.115162 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 01:53:17.115603 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:17.115232 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 01:53:17.200734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:17.200480 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:17.871741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:17.865360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:18.115414 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.115372 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 01:53:18.115843 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.115430 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 01:53:18.637883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.637849 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs"] Oct 31 01:53:18.644536 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.644507 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.661011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.660985 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5f65e1-serving-cert\"" Oct 31 01:53:18.663345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.663324 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs"] Oct 31 01:53:18.727477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.727230 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.727477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.727323 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-metrics-certs\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.727477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.727394 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cde4bb0-f471-4abd-8f34-00bbe0397607-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.727477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.727462 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4pkc\" (UniqueName: \"kubernetes.io/projected/3cde4bb0-f471-4abd-8f34-00bbe0397607-kube-api-access-c4pkc\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.788741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.788240 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:18.829007 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.828967 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.829225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.829061 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-metrics-certs\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.829225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.829122 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cde4bb0-f471-4abd-8f34-00bbe0397607-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.829225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.829194 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4pkc\" (UniqueName: \"kubernetes.io/projected/3cde4bb0-f471-4abd-8f34-00bbe0397607-kube-api-access-c4pkc\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.829992 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.829966 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cde4bb0-f471-4abd-8f34-00bbe0397607-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.832327 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.832299 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.832638 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.832616 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-metrics-certs\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.841643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.841617 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4pkc\" (UniqueName: \"kubernetes.io/projected/3cde4bb0-f471-4abd-8f34-00bbe0397607-kube-api-access-c4pkc\") pod \"maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:18.946998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.946919 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:18.988263 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:18.988233 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:19.115739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.115438 5090 patch_prober.go:29] interesting pod/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 31 01:53:19.115739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.115524 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 31 01:53:19.418510 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.418463 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs"] Oct 31 01:53:19.767740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.759738 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:19.961954 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.961916 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" event={"ID":"3cde4bb0-f471-4abd-8f34-00bbe0397607","Type":"ContainerStarted","Data":"45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f"} Oct 31 01:53:19.962131 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.961981 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" event={"ID":"3cde4bb0-f471-4abd-8f34-00bbe0397607","Type":"ContainerStarted","Data":"1a10f2217197d8d56efca2a99d9c869adeb2f22b7846c671033d96491cec2012"} Oct 31 01:53:19.962131 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.962010 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:19.985666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:19.985621 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" podStartSLOduration=1.537216913 podStartE2EDuration="1.985608788s" totalImagesPullingTime="448.391875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:19.983027292 +0000 UTC m=+2000.363272556" watchObservedRunningTime="2026-10-31 01:53:19.985608788 +0000 UTC m=+2000.365854042" Oct 31 01:53:20.146868 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:20.146629 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:20.147325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:20.147298 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:20.147901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:20.147881 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:53:20.560367 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:20.560327 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:21.342925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:21.342835 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:22.050863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:22.050772 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:22.752588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:22.752545 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:23.425305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:23.425273 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:24.070910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:24.070819 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:24.853633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:24.853593 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:25.552538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:25.552485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:25.975421 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:25.975344 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:53:26.188202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:26.188168 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:26.939993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:26.939953 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:27.634932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:27.634896 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:28.394616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:28.391101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:28.837205 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:28.837170 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v"] Oct 31 01:53:28.837801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:28.837765 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" podUID="4cab0118-dff0-4392-b6c4-06c7d7619070" containerName="maas-api" containerID="cri-o://102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e" gracePeriod=30 Oct 31 01:53:28.929761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:28.929702 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:29.090379 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.090298 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:29.313435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.313226 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:29.377546 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.377235 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:53:29.461774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.460221 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cab0118-dff0-4392-b6c4-06c7d7619070-kube-api-access-scp7v\" (UniqueName: \"kubernetes.io/projected/4cab0118-dff0-4392-b6c4-06c7d7619070-kube-api-access-scp7v\") pod \"4cab0118-dff0-4392-b6c4-06c7d7619070\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " Oct 31 01:53:29.461774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.460317 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cab0118-dff0-4392-b6c4-06c7d7619070-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab0118-dff0-4392-b6c4-06c7d7619070-openshift-service-ca\") pod \"4cab0118-dff0-4392-b6c4-06c7d7619070\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " Oct 31 01:53:29.461774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.460398 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-maas-api-tls\") pod \"4cab0118-dff0-4392-b6c4-06c7d7619070\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " Oct 31 01:53:29.461774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.460598 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-metrics-certs\") pod \"4cab0118-dff0-4392-b6c4-06c7d7619070\" (UID: \"4cab0118-dff0-4392-b6c4-06c7d7619070\") " Oct 31 01:53:29.476757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.474444 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cab0118-dff0-4392-b6c4-06c7d7619070-openshift-service-ca" pod "4cab0118-dff0-4392-b6c4-06c7d7619070" (UID: "4cab0118-dff0-4392-b6c4-06c7d7619070"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:29.504853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.485112 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:29.504853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.494871 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-metrics-certs" pod "4cab0118-dff0-4392-b6c4-06c7d7619070" (UID: "4cab0118-dff0-4392-b6c4-06c7d7619070"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:53:29.520789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.512905 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cab0118-dff0-4392-b6c4-06c7d7619070-kube-api-access-scp7v" pod "4cab0118-dff0-4392-b6c4-06c7d7619070" (UID: "4cab0118-dff0-4392-b6c4-06c7d7619070"). InnerVolumeSpecName "kube-api-access-scp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:29.520789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.519838 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-maas-api-tls" pod "4cab0118-dff0-4392-b6c4-06c7d7619070" (UID: "4cab0118-dff0-4392-b6c4-06c7d7619070"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:53:29.570203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.568985 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:29.570203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.569043 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cab0118-dff0-4392-b6c4-06c7d7619070-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:29.570203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.569059 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scp7v\" (UniqueName: \"kubernetes.io/projected/4cab0118-dff0-4392-b6c4-06c7d7619070-kube-api-access-scp7v\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:29.570203 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.569075 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cab0118-dff0-4392-b6c4-06c7d7619070-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:29.684161 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.682615 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:29.848134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:29.848101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:30.040972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.040934 5090 generic.go:356] "Generic (PLEG): container finished" podID="4cab0118-dff0-4392-b6c4-06c7d7619070" containerID="102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e" exitCode=0 Oct 31 01:53:30.041196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.040988 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" Oct 31 01:53:30.041196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.041025 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" event={"ID":"4cab0118-dff0-4392-b6c4-06c7d7619070","Type":"ContainerDied","Data":"102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e"} Oct 31 01:53:30.041196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.041085 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v" event={"ID":"4cab0118-dff0-4392-b6c4-06c7d7619070","Type":"ContainerDied","Data":"4f2350ee01cb274dc7400de9dc6cb6fa7b38e05ad00dc67ec185c8bcb41c6bb9"} Oct 31 01:53:30.041196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.041117 5090 scope.go:122] "RemoveContainer" containerID="102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e" Oct 31 01:53:30.051802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.051775 5090 scope.go:122] "RemoveContainer" containerID="102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e" Oct 31 01:53:30.052107 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:53:30.052083 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e\": container with ID starting with 102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e not found: ID does not exist" containerID="102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e" Oct 31 01:53:30.052191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.052119 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e"} err="failed to get container status \"102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e\": rpc error: code = NotFound desc = could not find container \"102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e\": container with ID starting with 102c84015abde3e8ad2ca1714eb1d386d7efa73651632fe403fe9f581a9dbb7e not found: ID does not exist" Oct 31 01:53:30.064985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.064953 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v"] Oct 31 01:53:30.069204 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.069180 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-a779b3-69c7cc457c-dg68v"] Oct 31 01:53:30.317144 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.317074 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:30.473861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.473817 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:30.879389 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:30.879314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:31.120904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:31.120867 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:31.797739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:31.787958 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cab0118-dff0-4392-b6c4-06c7d7619070" path="/var/lib/kubelet/pods/4cab0118-dff0-4392-b6c4-06c7d7619070/volumes" Oct 31 01:53:32.269740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:32.269608 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:34.134776 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:34.134206 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:34.611743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:34.607783 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:53:34.611743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:34.607861 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:53:34.691812 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:34.665073 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz"] Oct 31 01:53:34.834745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:34.834695 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:34.903512 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:53:34.903393 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:53:35.141501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:35.141468 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:36.070063 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:36.070032 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:37.049825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:37.049769 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:37.553407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:37.553362 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:37.696930 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:37.696898 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:37.846689 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:37.846600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:38.050840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:38.050807 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:38.853872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:38.853836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:39.102697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:39.102667 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:39.851238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:39.851026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:40.506012 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:40.505088 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:40.899081 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:40.899007 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg"] Oct 31 01:53:40.899244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:40.899191 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" containerID="cri-o://7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1" gracePeriod=30 Oct 31 01:53:41.358192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:41.357834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:42.201676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:42.201615 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:42.870788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:42.870755 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:43.507108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:43.507072 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:44.285293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.285252 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6"] Oct 31 01:53:44.286298 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.286271 5090 cpu_manager.go:414] "containerMap: removing container" podUID="4cab0118-dff0-4392-b6c4-06c7d7619070" containerName="maas-api" Oct 31 01:53:44.286444 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.286431 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cab0118-dff0-4392-b6c4-06c7d7619070" containerName="maas-api" Oct 31 01:53:44.286737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.286698 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cab0118-dff0-4392-b6c4-06c7d7619070" containerName="maas-api" Oct 31 01:53:44.292009 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.291990 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.310216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.310192 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6"] Oct 31 01:53:44.352839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.352812 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-77e2b5-serving-cert\"" Oct 31 01:53:44.382970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.382939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:44.428516 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.428486 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.428633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.428523 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cg9\" (UniqueName: \"kubernetes.io/projected/dd28a551-a279-436b-a4f6-4601ebfc400b-kube-api-access-67cg9\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.428633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.428604 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-metrics-certs\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.428749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.428654 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd28a551-a279-436b-a4f6-4601ebfc400b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.528777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.528114 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:44.532920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.530296 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.532920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.530348 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cg9\" (UniqueName: \"kubernetes.io/projected/dd28a551-a279-436b-a4f6-4601ebfc400b-kube-api-access-67cg9\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.532920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.530403 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-metrics-certs\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.532920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.530457 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd28a551-a279-436b-a4f6-4601ebfc400b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.532920 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.531559 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd28a551-a279-436b-a4f6-4601ebfc400b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.541787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.537964 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.546290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.546254 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-metrics-certs\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.549315 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.549286 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cg9\" (UniqueName: \"kubernetes.io/projected/dd28a551-a279-436b-a4f6-4601ebfc400b-kube-api-access-67cg9\") pod \"maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.617016 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.616989 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:44.950804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:44.948203 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6"] Oct 31 01:53:45.260747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:45.255058 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:45.328439 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:45.328405 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" event={"ID":"dd28a551-a279-436b-a4f6-4601ebfc400b","Type":"ContainerStarted","Data":"3a85db26b3273645910cff515138f16d9310f32286f46b5a8f308d681113444b"} Oct 31 01:53:45.952738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:45.952265 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:46.192516 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.192490 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:53:46.341640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.341602 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" event={"ID":"dd28a551-a279-436b-a4f6-4601ebfc400b","Type":"ContainerStarted","Data":"8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6"} Oct 31 01:53:46.341842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.341672 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:46.342983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.342962 5090 generic.go:356] "Generic (PLEG): container finished" podID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerID="7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1" exitCode=0 Oct 31 01:53:46.343095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.342994 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" Oct 31 01:53:46.343095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.343039 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" event={"ID":"e01ed085-5ab8-4f9e-ab11-5d40f3c14652","Type":"ContainerDied","Data":"7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1"} Oct 31 01:53:46.343095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.343084 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg" event={"ID":"e01ed085-5ab8-4f9e-ab11-5d40f3c14652","Type":"ContainerDied","Data":"2270584c8c7883cafc0f3c102f9468e7ed271240df3d739e715e77e0c0f89a22"} Oct 31 01:53:46.343199 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.343106 5090 scope.go:122] "RemoveContainer" containerID="7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1" Oct 31 01:53:46.354093 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.354078 5090 scope.go:122] "RemoveContainer" containerID="7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1" Oct 31 01:53:46.354382 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:53:46.354365 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1\": container with ID starting with 7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1 not found: ID does not exist" containerID="7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1" Oct 31 01:53:46.354462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.354397 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1"} err="failed to get container status \"7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1\": rpc error: code = NotFound desc = could not find container \"7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1\": container with ID starting with 7f700331025501f826e65ceac882104ff9589186dd61bee1cab9ce95f5a889a1 not found: ID does not exist" Oct 31 01:53:46.367756 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.367688 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" podStartSLOduration=1.926852458 podStartE2EDuration="2.367676008s" totalImagesPullingTime="440.82355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:53:46.365550721 +0000 UTC m=+2026.745795963" watchObservedRunningTime="2026-10-31 01:53:46.367676008 +0000 UTC m=+2026.747921258" Oct 31 01:53:46.378885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.378863 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-data\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.378996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.378904 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-credential-socket\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.378996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.378987 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-certs\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.379121 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379100 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-token\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-token\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.379181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379138 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-data" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:46.379181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379156 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-socket\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.379291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379195 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-podinfo\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.379383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379365 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-envoy\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.379445 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379394 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istiod-ca-cert\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.379445 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379411 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-kube-api-access-4nt7b\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-kube-api-access-4nt7b\") pod \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\" (UID: \"e01ed085-5ab8-4f9e-ab11-5d40f3c14652\") " Oct 31 01:53:46.379445 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379423 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-socket" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:46.379880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379866 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.379938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.379886 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.381544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.381525 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-token" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:46.381655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.381633 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-certs" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:46.382238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.382162 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istiod-ca-cert" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:46.382516 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.382481 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-credential-socket" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:46.383680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.383659 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-podinfo" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:53:46.383977 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.383959 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-envoy" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:46.384188 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.384161 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-kube-api-access-4nt7b" pod "e01ed085-5ab8-4f9e-ab11-5d40f3c14652" (UID: "e01ed085-5ab8-4f9e-ab11-5d40f3c14652"). InnerVolumeSpecName "kube-api-access-4nt7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:46.480996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.480916 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.480996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.480948 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.480996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.480963 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nt7b\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-kube-api-access-4nt7b\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.480996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.480974 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.480996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.480982 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.480996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.480990 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.480996 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.480998 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01ed085-5ab8-4f9e-ab11-5d40f3c14652-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:46.653420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.653335 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:46.693588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.693551 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg"] Oct 31 01:53:46.698995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:46.698957 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-6xwsg"] Oct 31 01:53:47.309735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:47.309683 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:47.748284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:47.748246 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" path="/var/lib/kubelet/pods/e01ed085-5ab8-4f9e-ab11-5d40f3c14652/volumes" Oct 31 01:53:48.025595 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:48.025516 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:48.660186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:48.660153 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:49.390970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:49.390930 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:49.793140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:49.793099 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" podUID="826a4006-be1b-435c-b8c8-f65b5b6c548c" containerName="main" containerID="cri-o://afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e" gracePeriod=45 Oct 31 01:53:50.079348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.079322 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:53:50.141253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141215 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/826a4006-be1b-435c-b8c8-f65b5b6c548c-tls-certs\" (UniqueName: \"kubernetes.io/secret/826a4006-be1b-435c-b8c8-f65b5b6c548c-tls-certs\") pod \"826a4006-be1b-435c-b8c8-f65b5b6c548c\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " Oct 31 01:53:50.141442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141335 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-tmp-dir\") pod \"826a4006-be1b-435c-b8c8-f65b5b6c548c\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " Oct 31 01:53:50.141442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141360 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826a4006-be1b-435c-b8c8-f65b5b6c548c-kube-api-access-4zgzn\" (UniqueName: \"kubernetes.io/projected/826a4006-be1b-435c-b8c8-f65b5b6c548c-kube-api-access-4zgzn\") pod \"826a4006-be1b-435c-b8c8-f65b5b6c548c\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " Oct 31 01:53:50.141442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141380 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-model-cache\") pod \"826a4006-be1b-435c-b8c8-f65b5b6c548c\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " Oct 31 01:53:50.141442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141406 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-home\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-home\") pod \"826a4006-be1b-435c-b8c8-f65b5b6c548c\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " Oct 31 01:53:50.142880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141594 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-tmp-dir" pod "826a4006-be1b-435c-b8c8-f65b5b6c548c" (UID: "826a4006-be1b-435c-b8c8-f65b5b6c548c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:50.142880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.142008 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-dshm\") pod \"826a4006-be1b-435c-b8c8-f65b5b6c548c\" (UID: \"826a4006-be1b-435c-b8c8-f65b5b6c548c\") " Oct 31 01:53:50.142880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141779 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-home" pod "826a4006-be1b-435c-b8c8-f65b5b6c548c" (UID: "826a4006-be1b-435c-b8c8-f65b5b6c548c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:50.142880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.141820 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-model-cache" pod "826a4006-be1b-435c-b8c8-f65b5b6c548c" (UID: "826a4006-be1b-435c-b8c8-f65b5b6c548c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:50.142880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.142788 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:50.142880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.142809 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:50.142880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.142822 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:50.145261 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.145230 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/826a4006-be1b-435c-b8c8-f65b5b6c548c-tls-certs" pod "826a4006-be1b-435c-b8c8-f65b5b6c548c" (UID: "826a4006-be1b-435c-b8c8-f65b5b6c548c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:53:50.145697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.145674 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826a4006-be1b-435c-b8c8-f65b5b6c548c-kube-api-access-4zgzn" pod "826a4006-be1b-435c-b8c8-f65b5b6c548c" (UID: "826a4006-be1b-435c-b8c8-f65b5b6c548c"). InnerVolumeSpecName "kube-api-access-4zgzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:50.146504 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.146483 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-dshm" pod "826a4006-be1b-435c-b8c8-f65b5b6c548c" (UID: "826a4006-be1b-435c-b8c8-f65b5b6c548c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:53:50.157504 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.157464 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:50.247686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.247649 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/826a4006-be1b-435c-b8c8-f65b5b6c548c-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:50.247686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.247686 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/826a4006-be1b-435c-b8c8-f65b5b6c548c-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:50.247986 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.247701 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zgzn\" (UniqueName: \"kubernetes.io/projected/826a4006-be1b-435c-b8c8-f65b5b6c548c-kube-api-access-4zgzn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:50.376026 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.375937 5090 generic.go:356] "Generic (PLEG): container finished" podID="826a4006-be1b-435c-b8c8-f65b5b6c548c" containerID="afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e" exitCode=0 Oct 31 01:53:50.376026 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.376013 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" Oct 31 01:53:50.376238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.376028 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" event={"ID":"826a4006-be1b-435c-b8c8-f65b5b6c548c","Type":"ContainerDied","Data":"afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e"} Oct 31 01:53:50.376238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.376083 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz" event={"ID":"826a4006-be1b-435c-b8c8-f65b5b6c548c","Type":"ContainerDied","Data":"3ba898135f5bdb7b46ca572b98666e21c58899f874336b30b13d8fb3b56796d4"} Oct 31 01:53:50.376238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.376108 5090 scope.go:122] "RemoveContainer" containerID="afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e" Oct 31 01:53:50.388469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.388449 5090 scope.go:122] "RemoveContainer" containerID="afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e" Oct 31 01:53:50.388756 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:53:50.388737 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e\": container with ID starting with afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e not found: ID does not exist" containerID="afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e" Oct 31 01:53:50.388832 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.388765 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e"} err="failed to get container status \"afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e\": rpc error: code = NotFound desc = could not find container \"afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e\": container with ID starting with afa3c8e332f8b48fed9ffa333be8d972c209595122ba07c2ff4b3f405ff3da5e not found: ID does not exist" Oct 31 01:53:50.400707 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.400685 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz"] Oct 31 01:53:50.405978 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.405955 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-a779b3/praxis-route-a779b3-kserve-5588676559-d2fsz"] Oct 31 01:53:50.685361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.685235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:50.849559 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:50.849515 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:51.590846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:51.590814 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:51.747589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:51.747558 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826a4006-be1b-435c-b8c8-f65b5b6c548c" path="/var/lib/kubelet/pods/826a4006-be1b-435c-b8c8-f65b5b6c548c/volumes" Oct 31 01:53:52.243739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:52.242695 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:52.357824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:52.357798 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:53:52.981369 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:52.980657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:53.589557 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:53.589512 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:54.257741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:54.252872 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:55.137538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:55.137507 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:55.594582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:55.594528 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq"] Oct 31 01:53:55.595108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:55.595057 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" podUID="ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" containerName="maas-api" containerID="cri-o://276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2" gracePeriod=30 Oct 31 01:53:55.698292 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:55.698252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:55.904793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:55.904527 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:56.041776 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.033116 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:53:56.058402 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.058302 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:56.201924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.193927 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-maas-api-tls\") pod \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " Oct 31 01:53:56.201924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.194040 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-openshift-service-ca\") pod \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " Oct 31 01:53:56.201924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.194097 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-kube-api-access-stp6b\" (UniqueName: \"kubernetes.io/projected/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-kube-api-access-stp6b\") pod \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " Oct 31 01:53:56.201924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.194154 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-metrics-certs\") pod \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\" (UID: \"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f\") " Oct 31 01:53:56.201924 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.197167 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-openshift-service-ca" pod "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" (UID: "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:53:56.228000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.227066 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-kube-api-access-stp6b" pod "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" (UID: "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f"). InnerVolumeSpecName "kube-api-access-stp6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:53:56.229787 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.229588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:56.234839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.234809 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-maas-api-tls" pod "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" (UID: "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:53:56.265348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.264345 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-metrics-certs" pod "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" (UID: "ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:53:56.389624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.359079 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stp6b\" (UniqueName: \"kubernetes.io/projected/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-kube-api-access-stp6b\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:56.389624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.359122 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:56.389624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.359139 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:56.389624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.359154 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:53:56.472853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.469183 5090 generic.go:356] "Generic (PLEG): container finished" podID="ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" containerID="276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2" exitCode=0 Oct 31 01:53:56.472853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.469244 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" Oct 31 01:53:56.472853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.469322 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" event={"ID":"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f","Type":"ContainerDied","Data":"276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2"} Oct 31 01:53:56.472853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.469362 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq" event={"ID":"ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f","Type":"ContainerDied","Data":"dd80a5ee349f65c476c316d29591b9d6e9556c4c2b3d05d5183d61e594906faa"} Oct 31 01:53:56.472853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.469389 5090 scope.go:122] "RemoveContainer" containerID="276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2" Oct 31 01:53:56.527911 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.527878 5090 scope.go:122] "RemoveContainer" containerID="276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2" Oct 31 01:53:56.536292 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:53:56.536158 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2\": container with ID starting with 276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2 not found: ID does not exist" containerID="276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2" Oct 31 01:53:56.536292 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.536207 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2"} err="failed to get container status \"276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2\": rpc error: code = NotFound desc = could not find container \"276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2\": container with ID starting with 276f3fbc2c39a552eceecd11c118b980b4a3d41f9ed34089813650ea19b8eba2 not found: ID does not exist" Oct 31 01:53:56.549343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.549183 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:56.558980 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.556770 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq"] Oct 31 01:53:56.581737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:56.578877 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-925bac-7bb69dd499-6mjnq"] Oct 31 01:53:57.813503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:57.813425 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" path="/var/lib/kubelet/pods/ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f/volumes" Oct 31 01:53:59.074860 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:59.074787 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:53:59.766000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:53:59.765424 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:00.657280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:00.657245 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:01.713808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:01.713772 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:02.457310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:02.457235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:02.824512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:02.824486 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:03.243921 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:03.243891 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:03.968484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:03.968451 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:04.140643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:04.140609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:04.423476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:04.423440 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6"] Oct 31 01:54:04.423794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:04.423700 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" containerID="cri-o://56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc" gracePeriod=30 Oct 31 01:54:04.469776 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:04.469750 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:04.794001 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:04.793961 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:54:05.173958 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:05.173858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:05.944348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:05.944313 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:06.135373 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:06.135342 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:06.659768 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:06.646909 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:06.967023 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:06.966876 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:07.172092 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:07.172010 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:08.003234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:08.003203 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:08.994102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:08.994070 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:09.744318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.744285 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:54:09.799601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.799569 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-podinfo\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.799782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.799615 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-kube-api-access-nd7np\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-kube-api-access-nd7np\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.799782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.799759 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-data\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.799912 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.799836 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-credential-socket\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.799912 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.799881 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-token\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-token\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.800010 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.799918 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istiod-ca-cert\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.800010 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.799941 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-envoy\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.800010 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.800007 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-certs\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.800567 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.800042 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-socket\") pod \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\" (UID: \"151f3d99-9efa-46a4-ac2a-7dd284ed2cea\") " Oct 31 01:54:09.802434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.802397 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-credential-socket" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:09.802676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.802651 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-data" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:09.804109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.803506 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-podinfo" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:54:09.804432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.804258 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-certs" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:09.804766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.804566 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-socket" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:09.805027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.804957 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istiod-ca-cert" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:54:09.805645 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.805620 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-envoy" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:09.808090 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.808053 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-token" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:54:09.808362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.808330 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-kube-api-access-nd7np" pod "151f3d99-9efa-46a4-ac2a-7dd284ed2cea" (UID: "151f3d99-9efa-46a4-ac2a-7dd284ed2cea"). InnerVolumeSpecName "kube-api-access-nd7np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:54:09.826609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.826032 5090 generic.go:356] "Generic (PLEG): container finished" podID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerID="56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc" exitCode=0 Oct 31 01:54:09.826609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.826106 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" Oct 31 01:54:09.826609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.826176 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" event={"ID":"151f3d99-9efa-46a4-ac2a-7dd284ed2cea","Type":"ContainerDied","Data":"56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc"} Oct 31 01:54:09.826609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.826224 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6" event={"ID":"151f3d99-9efa-46a4-ac2a-7dd284ed2cea","Type":"ContainerDied","Data":"3e2c66cd0009eb641396971513d64ee884e4c3b5308931e3f0adc130772c1037"} Oct 31 01:54:09.826609 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.826252 5090 scope.go:122] "RemoveContainer" containerID="56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc" Oct 31 01:54:09.842108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.842086 5090 scope.go:122] "RemoveContainer" containerID="56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc" Oct 31 01:54:09.842497 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:09.842466 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc\": container with ID starting with 56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc not found: ID does not exist" containerID="56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc" Oct 31 01:54:09.842591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.842505 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc"} err="failed to get container status \"56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc\": rpc error: code = NotFound desc = could not find container \"56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc\": container with ID starting with 56273ac116e2ad6fe4f7d7f377331893362be85b3351820cbdccece35c703bcc not found: ID does not exist" Oct 31 01:54:09.855642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.855610 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6"] Oct 31 01:54:09.866268 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.866234 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-925bac-openshift-default-6846b8897c-8spj6"] Oct 31 01:54:09.902455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902418 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902447 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902463 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902477 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902491 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902505 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902513 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902521 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:09.902615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:09.902530 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd7np\" (UniqueName: \"kubernetes.io/projected/151f3d99-9efa-46a4-ac2a-7dd284ed2cea-kube-api-access-nd7np\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:10.064277 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:10.063459 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:10.241514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:10.241478 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:10.391872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:10.391770 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:11.089585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:11.089533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:11.559040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:11.559011 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:11.788514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:11.768635 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:11.799283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:11.799252 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" path="/var/lib/kubelet/pods/151f3d99-9efa-46a4-ac2a-7dd284ed2cea/volumes" Oct 31 01:54:12.159876 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:12.158081 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:12.930700 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:12.930671 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:13.075892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:13.075854 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:13.609000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:13.608963 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:13.929511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:13.929415 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:14.433771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:14.433742 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:14.587089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:14.586501 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:14.812531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:14.812499 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:15.680708 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:15.680675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:16.269501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.269467 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs"] Oct 31 01:54:16.270049 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.269735 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" podUID="3cde4bb0-f471-4abd-8f34-00bbe0397607" containerName="maas-api" containerID="cri-o://45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f" gracePeriod=30 Oct 31 01:54:16.346596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.346563 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:16.485670 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.483517 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:16.698175 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.688320 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:16.782765 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.778910 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:54:16.843403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.843330 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:16.890905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.890799 5090 generic.go:356] "Generic (PLEG): container finished" podID="3cde4bb0-f471-4abd-8f34-00bbe0397607" containerID="45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f" exitCode=0 Oct 31 01:54:16.890905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.890892 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" event={"ID":"3cde4bb0-f471-4abd-8f34-00bbe0397607","Type":"ContainerDied","Data":"45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f"} Oct 31 01:54:16.891154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.890947 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" event={"ID":"3cde4bb0-f471-4abd-8f34-00bbe0397607","Type":"ContainerDied","Data":"1a10f2217197d8d56efca2a99d9c869adeb2f22b7846c671033d96491cec2012"} Oct 31 01:54:16.891154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.890981 5090 scope.go:122] "RemoveContainer" containerID="45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f" Oct 31 01:54:16.891154 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.891132 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs" Oct 31 01:54:16.904043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.903396 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cde4bb0-f471-4abd-8f34-00bbe0397607-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cde4bb0-f471-4abd-8f34-00bbe0397607-openshift-service-ca\") pod \"3cde4bb0-f471-4abd-8f34-00bbe0397607\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " Oct 31 01:54:16.904043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.903468 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cde4bb0-f471-4abd-8f34-00bbe0397607-kube-api-access-c4pkc\" (UniqueName: \"kubernetes.io/projected/3cde4bb0-f471-4abd-8f34-00bbe0397607-kube-api-access-c4pkc\") pod \"3cde4bb0-f471-4abd-8f34-00bbe0397607\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " Oct 31 01:54:16.904043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.903608 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-maas-api-tls\") pod \"3cde4bb0-f471-4abd-8f34-00bbe0397607\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " Oct 31 01:54:16.904043 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.903682 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-metrics-certs\") pod \"3cde4bb0-f471-4abd-8f34-00bbe0397607\" (UID: \"3cde4bb0-f471-4abd-8f34-00bbe0397607\") " Oct 31 01:54:16.905585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.905556 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cde4bb0-f471-4abd-8f34-00bbe0397607-openshift-service-ca" pod "3cde4bb0-f471-4abd-8f34-00bbe0397607" (UID: "3cde4bb0-f471-4abd-8f34-00bbe0397607"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:54:16.907257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.907188 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-metrics-certs" pod "3cde4bb0-f471-4abd-8f34-00bbe0397607" (UID: "3cde4bb0-f471-4abd-8f34-00bbe0397607"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:54:16.908220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.908186 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-maas-api-tls" pod "3cde4bb0-f471-4abd-8f34-00bbe0397607" (UID: "3cde4bb0-f471-4abd-8f34-00bbe0397607"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:54:16.910880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.909769 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cde4bb0-f471-4abd-8f34-00bbe0397607-kube-api-access-c4pkc" pod "3cde4bb0-f471-4abd-8f34-00bbe0397607" (UID: "3cde4bb0-f471-4abd-8f34-00bbe0397607"). InnerVolumeSpecName "kube-api-access-c4pkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:54:16.918428 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.918401 5090 scope.go:122] "RemoveContainer" containerID="45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f" Oct 31 01:54:16.919354 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:16.919327 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f\": container with ID starting with 45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f not found: ID does not exist" containerID="45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f" Oct 31 01:54:16.919423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:16.919370 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f"} err="failed to get container status \"45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f\": rpc error: code = NotFound desc = could not find container \"45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f\": container with ID starting with 45eaaef3381b126a78bcb26047cc56d3d5aaa35a2b84aa20728eda1f0d07aa3f not found: ID does not exist" Oct 31 01:54:17.005579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.005546 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cde4bb0-f471-4abd-8f34-00bbe0397607-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:17.005579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.005578 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4pkc\" (UniqueName: \"kubernetes.io/projected/3cde4bb0-f471-4abd-8f34-00bbe0397607-kube-api-access-c4pkc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:17.005877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.005592 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:17.005877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.005604 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cde4bb0-f471-4abd-8f34-00bbe0397607-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:17.221060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.221024 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs"] Oct 31 01:54:17.225631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.225596 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5f65e1-7f4868d998-dcjjs"] Oct 31 01:54:17.226614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.226590 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:17.658794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.658759 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6"] Oct 31 01:54:17.662371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662346 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" Oct 31 01:54:17.662371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662371 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662390 5090 cpu_manager.go:414] "containerMap: removing container" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662398 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662409 5090 cpu_manager.go:414] "containerMap: removing container" podUID="3cde4bb0-f471-4abd-8f34-00bbe0397607" containerName="maas-api" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662415 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cde4bb0-f471-4abd-8f34-00bbe0397607" containerName="maas-api" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662448 5090 cpu_manager.go:414] "containerMap: removing container" podUID="ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" containerName="maas-api" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662455 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" containerName="maas-api" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662483 5090 cpu_manager.go:414] "containerMap: removing container" podUID="826a4006-be1b-435c-b8c8-f65b5b6c548c" containerName="main" Oct 31 01:54:17.662554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662491 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826a4006-be1b-435c-b8c8-f65b5b6c548c" containerName="main" Oct 31 01:54:17.662923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662640 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="151f3d99-9efa-46a4-ac2a-7dd284ed2cea" containerName="istio-proxy" Oct 31 01:54:17.662923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662654 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cde4bb0-f471-4abd-8f34-00bbe0397607" containerName="maas-api" Oct 31 01:54:17.662923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662667 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec37f641-a7d2-4bf0-b8b2-4e37ff4c6a8f" containerName="maas-api" Oct 31 01:54:17.662923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662680 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="826a4006-be1b-435c-b8c8-f65b5b6c548c" containerName="main" Oct 31 01:54:17.662923 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.662698 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01ed085-5ab8-4f9e-ab11-5d40f3c14652" containerName="istio-proxy" Oct 31 01:54:17.668410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.668382 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.679347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.679316 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6"] Oct 31 01:54:17.686174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.685985 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-eb8ae6-serving-cert\"" Oct 31 01:54:17.712995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.712968 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.713122 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.713004 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.713256 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.713239 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ebf99-38fa-43f9-8ccf-d1968397d926-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.713302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.713289 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6nq\" (UniqueName: \"kubernetes.io/projected/d10ebf99-38fa-43f9-8ccf-d1968397d926-kube-api-access-qr6nq\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.748420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.748389 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cde4bb0-f471-4abd-8f34-00bbe0397607" path="/var/lib/kubelet/pods/3cde4bb0-f471-4abd-8f34-00bbe0397607/volumes" Oct 31 01:54:17.748693 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.748673 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:17.816736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.815963 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6nq\" (UniqueName: \"kubernetes.io/projected/d10ebf99-38fa-43f9-8ccf-d1968397d926-kube-api-access-qr6nq\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.816736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.816144 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.816736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.816176 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.816736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.816353 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ebf99-38fa-43f9-8ccf-d1968397d926-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.819017 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.818987 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ebf99-38fa-43f9-8ccf-d1968397d926-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.823397 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.823348 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.825481 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.824888 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.828618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.828577 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6nq\" (UniqueName: \"kubernetes.io/projected/d10ebf99-38fa-43f9-8ccf-d1968397d926-kube-api-access-qr6nq\") pod \"maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:17.899372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.899340 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:17.997631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:17.997600 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:18.367781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:18.356969 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6"] Oct 31 01:54:18.666631 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:18.665034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:18.908196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:18.908171 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" event={"ID":"d10ebf99-38fa-43f9-8ccf-d1968397d926","Type":"ContainerStarted","Data":"697e50129e1d26429443129f0cdfb3894767f186224e3968e9764b6e2e799c23"} Oct 31 01:54:18.908319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:18.908224 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" event={"ID":"d10ebf99-38fa-43f9-8ccf-d1968397d926","Type":"ContainerStarted","Data":"dae9a5c08469b4500baa2911afdce44a26ab1bad73e65dc272dcb96fb435412d"} Oct 31 01:54:18.908373 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:18.908324 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:18.930770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:18.930659 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" podStartSLOduration=1.490341318 podStartE2EDuration="1.930646308s" totalImagesPullingTime="440.30499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:18.928788626 +0000 UTC m=+2059.309033866" watchObservedRunningTime="2026-10-31 01:54:18.930646308 +0000 UTC m=+2059.310891569" Oct 31 01:54:19.222374 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:19.222303 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:19.542248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:19.542214 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:20.206403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:20.206370 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:20.350623 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:20.350591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:20.527353 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:20.527318 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74"] Oct 31 01:54:20.527610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:20.527579 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" containerID="cri-o://f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378" gracePeriod=30 Oct 31 01:54:20.591701 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:20.591669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:21.068850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:21.068815 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:21.874771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:21.873117 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:22.572904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:22.572863 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:23.346275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:23.346239 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:24.068909 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:24.068871 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:24.216930 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:24.216890 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:24.486611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:24.486524 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:24.922306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:24.922274 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:54:24.935393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:24.935364 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:25.914472 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.914443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:25.923126 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.923105 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:54:25.974738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974325 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-podinfo\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.974738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974432 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-envoy\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.974738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974482 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-token\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-token\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.974738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974558 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-credential-socket\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.974738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974598 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deddf906-2736-4ef8-8c29-33e2a81bc256-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deddf906-2736-4ef8-8c29-33e2a81bc256-istiod-ca-cert\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.974738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974630 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-socket\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.974738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974697 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-certs\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.975286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974784 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-kube-api-access-n5796\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-kube-api-access-n5796\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.975286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.974818 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-data\") pod \"deddf906-2736-4ef8-8c29-33e2a81bc256\" (UID: \"deddf906-2736-4ef8-8c29-33e2a81bc256\") " Oct 31 01:54:25.977429 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.977345 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-data" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:25.980671 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.980643 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deddf906-2736-4ef8-8c29-33e2a81bc256-istiod-ca-cert" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:54:25.981408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.981382 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-certs" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:25.981776 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.981752 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-socket" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:25.982106 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.982084 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-credential-socket" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:25.992850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.992801 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-podinfo" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:54:25.995208 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.995160 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-envoy" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:54:25.995952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:25.995916 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-token" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:54:26.002797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.002745 5090 generic.go:356] "Generic (PLEG): container finished" podID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerID="f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378" exitCode=0 Oct 31 01:54:26.002899 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.002795 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" event={"ID":"deddf906-2736-4ef8-8c29-33e2a81bc256","Type":"ContainerDied","Data":"f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378"} Oct 31 01:54:26.002899 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.002844 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" event={"ID":"deddf906-2736-4ef8-8c29-33e2a81bc256","Type":"ContainerDied","Data":"6452e073ad14e833ead230e783c83a287babc45f73bf172ba4ec7d3d31eb4156"} Oct 31 01:54:26.002899 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.002874 5090 scope.go:122] "RemoveContainer" containerID="f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378" Oct 31 01:54:26.003068 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.003035 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74" Oct 31 01:54:26.003508 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.003469 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-kube-api-access-n5796" pod "deddf906-2736-4ef8-8c29-33e2a81bc256" (UID: "deddf906-2736-4ef8-8c29-33e2a81bc256"). InnerVolumeSpecName "kube-api-access-n5796". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:54:26.027749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.027691 5090 scope.go:122] "RemoveContainer" containerID="f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378" Oct 31 01:54:26.028215 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:26.028141 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378\": container with ID starting with f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378 not found: ID does not exist" containerID="f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378" Oct 31 01:54:26.028215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.028185 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378"} err="failed to get container status \"f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378\": rpc error: code = NotFound desc = could not find container \"f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378\": container with ID starting with f32a0f23f227c47571df2a96a8ea395d52ccd24b0d47964483b25b1583df5378 not found: ID does not exist" Oct 31 01:54:26.076824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.076637 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.076824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.076675 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deddf906-2736-4ef8-8c29-33e2a81bc256-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.076824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.076691 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.077293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.076706 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.077293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.077182 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5796\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-kube-api-access-n5796\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.077293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.077215 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.077293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.077231 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.077293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.077244 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.077293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.077257 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deddf906-2736-4ef8-8c29-33e2a81bc256-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:26.343330 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.343253 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74"] Oct 31 01:54:26.357007 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:26.356980 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-a779b3-openshift-default-5f9c9bb6df-znq74"] Oct 31 01:54:27.731345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:27.731310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:27.789745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:27.781509 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" path="/var/lib/kubelet/pods/deddf906-2736-4ef8-8c29-33e2a81bc256/volumes" Oct 31 01:54:29.223746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:29.223691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:30.183750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:30.183606 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:30.445781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:30.445659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:31.074558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:31.073972 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:31.714307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:31.714272 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:31.873871 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:31.873837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:32.341345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:32.335322 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:32.476300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:32.476270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:32.957331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:32.955680 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:33.650960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:33.650916 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:34.344880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:34.344845 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:34.790288 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:34.790243 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:54:35.068806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:35.059200 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:35.231860 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:35.231827 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:35.372206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:35.372130 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:36.118377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:36.118328 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:36.294437 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:36.294409 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:36.522470 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:36.522444 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:36.664163 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:36.664135 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:37.102597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:37.102532 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:37.470305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:37.470184 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:37.639769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:37.639646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:37.799325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:37.799291 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p"] Oct 31 01:54:37.799531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:37.799502 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" podUID="4eec0817-1add-4a0b-af21-3cbdf992c01a" containerName="maas-api" containerID="cri-o://13078e19d4efc1f9a715f00e8462352815aae5b852e7acef7292a6af9ffeec0c" gracePeriod=30 Oct 31 01:54:37.892652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:37.892600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:38.096423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.096229 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:38.251861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.251824 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:38.257371 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.257339 5090 generic.go:356] "Generic (PLEG): container finished" podID="4eec0817-1add-4a0b-af21-3cbdf992c01a" containerID="13078e19d4efc1f9a715f00e8462352815aae5b852e7acef7292a6af9ffeec0c" exitCode=0 Oct 31 01:54:38.257754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.257708 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" event={"ID":"4eec0817-1add-4a0b-af21-3cbdf992c01a","Type":"ContainerDied","Data":"13078e19d4efc1f9a715f00e8462352815aae5b852e7acef7292a6af9ffeec0c"} Oct 31 01:54:38.341647 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.341620 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:54:38.356361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.356303 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-metrics-certs\") pod \"4eec0817-1add-4a0b-af21-3cbdf992c01a\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " Oct 31 01:54:38.356485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.356417 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-maas-api-tls\") pod \"4eec0817-1add-4a0b-af21-3cbdf992c01a\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " Oct 31 01:54:38.356485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.356451 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eec0817-1add-4a0b-af21-3cbdf992c01a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eec0817-1add-4a0b-af21-3cbdf992c01a-openshift-service-ca\") pod \"4eec0817-1add-4a0b-af21-3cbdf992c01a\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " Oct 31 01:54:38.356556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.356537 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eec0817-1add-4a0b-af21-3cbdf992c01a-kube-api-access-zhr9x\" (UniqueName: \"kubernetes.io/projected/4eec0817-1add-4a0b-af21-3cbdf992c01a-kube-api-access-zhr9x\") pod \"4eec0817-1add-4a0b-af21-3cbdf992c01a\" (UID: \"4eec0817-1add-4a0b-af21-3cbdf992c01a\") " Oct 31 01:54:38.356902 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.356874 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eec0817-1add-4a0b-af21-3cbdf992c01a-openshift-service-ca" pod "4eec0817-1add-4a0b-af21-3cbdf992c01a" (UID: "4eec0817-1add-4a0b-af21-3cbdf992c01a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:54:38.357216 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.357141 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eec0817-1add-4a0b-af21-3cbdf992c01a-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:38.358523 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.358500 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-maas-api-tls" pod "4eec0817-1add-4a0b-af21-3cbdf992c01a" (UID: "4eec0817-1add-4a0b-af21-3cbdf992c01a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:54:38.358608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.358594 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-metrics-certs" pod "4eec0817-1add-4a0b-af21-3cbdf992c01a" (UID: "4eec0817-1add-4a0b-af21-3cbdf992c01a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:54:38.358815 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.358787 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eec0817-1add-4a0b-af21-3cbdf992c01a-kube-api-access-zhr9x" pod "4eec0817-1add-4a0b-af21-3cbdf992c01a" (UID: "4eec0817-1add-4a0b-af21-3cbdf992c01a"). InnerVolumeSpecName "kube-api-access-zhr9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:54:38.407368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.407327 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:38.458610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.458580 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:38.458610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.458607 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eec0817-1add-4a0b-af21-3cbdf992c01a-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:38.458838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.458617 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhr9x\" (UniqueName: \"kubernetes.io/projected/4eec0817-1add-4a0b-af21-3cbdf992c01a-kube-api-access-zhr9x\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:38.816837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:38.816804 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:39.272331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.272142 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:39.280758 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.280730 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" Oct 31 01:54:39.280964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.280779 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p" event={"ID":"4eec0817-1add-4a0b-af21-3cbdf992c01a","Type":"ContainerDied","Data":"22d45b202ab5b31617cca067fb3e229ad771d2b98fd335f7221cb5947013a654"} Oct 31 01:54:39.280964 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.280877 5090 scope.go:122] "RemoveContainer" containerID="13078e19d4efc1f9a715f00e8462352815aae5b852e7acef7292a6af9ffeec0c" Oct 31 01:54:39.321499 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.321468 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p"] Oct 31 01:54:39.328398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.328367 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-726dd1-54dfb8fcd5-c9z4p"] Oct 31 01:54:39.363808 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:39.355859 5090 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eec0817_1add_4a0b_af21_3cbdf992c01a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eec0817_1add_4a0b_af21_3cbdf992c01a.slice/crio-22d45b202ab5b31617cca067fb3e229ad771d2b98fd335f7221cb5947013a654\": RecentStats: unable to find data in memory cache]" Oct 31 01:54:39.748788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.748758 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eec0817-1add-4a0b-af21-3cbdf992c01a" path="/var/lib/kubelet/pods/4eec0817-1add-4a0b-af21-3cbdf992c01a/volumes" Oct 31 01:54:39.960746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:39.944809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:40.384500 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:40.384463 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6"] Oct 31 01:54:40.384761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:40.384704 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" podUID="dd28a551-a279-436b-a4f6-4601ebfc400b" containerName="maas-api" containerID="cri-o://8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6" gracePeriod=30 Oct 31 01:54:40.473465 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:40.473434 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:40.682728 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:40.682526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:40.847897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:40.847866 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:40.914168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:40.914006 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:54:41.016262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.016233 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd28a551-a279-436b-a4f6-4601ebfc400b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd28a551-a279-436b-a4f6-4601ebfc400b-openshift-service-ca\") pod \"dd28a551-a279-436b-a4f6-4601ebfc400b\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " Oct 31 01:54:41.016757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.016366 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-maas-api-tls\") pod \"dd28a551-a279-436b-a4f6-4601ebfc400b\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " Oct 31 01:54:41.016757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.016482 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd28a551-a279-436b-a4f6-4601ebfc400b-kube-api-access-67cg9\" (UniqueName: \"kubernetes.io/projected/dd28a551-a279-436b-a4f6-4601ebfc400b-kube-api-access-67cg9\") pod \"dd28a551-a279-436b-a4f6-4601ebfc400b\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " Oct 31 01:54:41.016757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.016503 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-metrics-certs\") pod \"dd28a551-a279-436b-a4f6-4601ebfc400b\" (UID: \"dd28a551-a279-436b-a4f6-4601ebfc400b\") " Oct 31 01:54:41.016757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.016582 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd28a551-a279-436b-a4f6-4601ebfc400b-openshift-service-ca" pod "dd28a551-a279-436b-a4f6-4601ebfc400b" (UID: "dd28a551-a279-436b-a4f6-4601ebfc400b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:54:41.016998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.016961 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd28a551-a279-436b-a4f6-4601ebfc400b-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:41.018851 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.018828 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd28a551-a279-436b-a4f6-4601ebfc400b-kube-api-access-67cg9" pod "dd28a551-a279-436b-a4f6-4601ebfc400b" (UID: "dd28a551-a279-436b-a4f6-4601ebfc400b"). InnerVolumeSpecName "kube-api-access-67cg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:54:41.018931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.018882 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-maas-api-tls" pod "dd28a551-a279-436b-a4f6-4601ebfc400b" (UID: "dd28a551-a279-436b-a4f6-4601ebfc400b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:54:41.018931 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.018901 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-metrics-certs" pod "dd28a551-a279-436b-a4f6-4601ebfc400b" (UID: "dd28a551-a279-436b-a4f6-4601ebfc400b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:54:41.036368 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.036343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:41.118751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.118699 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67cg9\" (UniqueName: \"kubernetes.io/projected/dd28a551-a279-436b-a4f6-4601ebfc400b-kube-api-access-67cg9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:41.118751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.118754 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:41.118965 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.118765 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd28a551-a279-436b-a4f6-4601ebfc400b-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:54:41.296416 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.296380 5090 generic.go:356] "Generic (PLEG): container finished" podID="dd28a551-a279-436b-a4f6-4601ebfc400b" containerID="8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6" exitCode=0 Oct 31 01:54:41.296571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.296435 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" Oct 31 01:54:41.296571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.296462 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" event={"ID":"dd28a551-a279-436b-a4f6-4601ebfc400b","Type":"ContainerDied","Data":"8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6"} Oct 31 01:54:41.296571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.296517 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6" event={"ID":"dd28a551-a279-436b-a4f6-4601ebfc400b","Type":"ContainerDied","Data":"3a85db26b3273645910cff515138f16d9310f32286f46b5a8f308d681113444b"} Oct 31 01:54:41.296571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.296542 5090 scope.go:122] "RemoveContainer" containerID="8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6" Oct 31 01:54:41.307624 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.307605 5090 scope.go:122] "RemoveContainer" containerID="8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6" Oct 31 01:54:41.308014 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:41.307996 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6\": container with ID starting with 8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6 not found: ID does not exist" containerID="8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6" Oct 31 01:54:41.308092 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.308023 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6"} err="failed to get container status \"8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6\": rpc error: code = NotFound desc = could not find container \"8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6\": container with ID starting with 8ea73597faa8f3c630deb3eb559316b44e0390eb374f1cd01062c40a7f7c6fd6 not found: ID does not exist" Oct 31 01:54:41.320290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.320271 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6"] Oct 31 01:54:41.325744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.325689 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-77e2b5-548fbfd65-c8kj6"] Oct 31 01:54:41.442064 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.442029 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:41.759304 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.759269 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd28a551-a279-436b-a4f6-4601ebfc400b" path="/var/lib/kubelet/pods/dd28a551-a279-436b-a4f6-4601ebfc400b/volumes" Oct 31 01:54:41.775769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:41.775703 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:42.355732 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.355677 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc"] Oct 31 01:54:42.356348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356303 5090 cpu_manager.go:414] "containerMap: removing container" podUID="4eec0817-1add-4a0b-af21-3cbdf992c01a" containerName="maas-api" Oct 31 01:54:42.356348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356319 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eec0817-1add-4a0b-af21-3cbdf992c01a" containerName="maas-api" Oct 31 01:54:42.356348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356332 5090 cpu_manager.go:414] "containerMap: removing container" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" Oct 31 01:54:42.356348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356337 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" Oct 31 01:54:42.356596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356357 5090 cpu_manager.go:414] "containerMap: removing container" podUID="dd28a551-a279-436b-a4f6-4601ebfc400b" containerName="maas-api" Oct 31 01:54:42.356596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356362 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd28a551-a279-436b-a4f6-4601ebfc400b" containerName="maas-api" Oct 31 01:54:42.356596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356444 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd28a551-a279-436b-a4f6-4601ebfc400b" containerName="maas-api" Oct 31 01:54:42.356596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356459 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eec0817-1add-4a0b-af21-3cbdf992c01a" containerName="maas-api" Oct 31 01:54:42.356596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.356466 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="deddf906-2736-4ef8-8c29-33e2a81bc256" containerName="istio-proxy" Oct 31 01:54:42.361062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.361040 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.370234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.369671 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-dbff33-serving-cert\"" Oct 31 01:54:42.370989 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.370966 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc"] Oct 31 01:54:42.434629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.434587 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.434814 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.434651 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-metrics-certs\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.434814 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.434767 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxvl\" (UniqueName: \"kubernetes.io/projected/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-kube-api-access-ggxvl\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.434900 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.434822 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.441088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.441066 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:42.536215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.536173 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxvl\" (UniqueName: \"kubernetes.io/projected/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-kube-api-access-ggxvl\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.536423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.536261 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.536423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.536403 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.536584 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:42.536450 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-serving-cert: secret "maas-api-e2e-praxis-b-dbff33-serving-cert" not found Oct 31 01:54:42.536584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.536509 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-metrics-certs\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.536584 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:54:42.536537 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls podName:5b241c23-8b2b-4aa7-879a-ad1d8fde83f9 nodeName:}" failed. No retries permitted until 2026-10-31 01:54:43.03651431 +0000 UTC m=+2083.416759527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls") pod "maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" (UID: "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9") : secret "maas-api-e2e-praxis-b-dbff33-serving-cert" not found Oct 31 01:54:42.537286 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.537262 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.539300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.539279 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-metrics-certs\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.555215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.555188 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxvl\" (UniqueName: \"kubernetes.io/projected/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-kube-api-access-ggxvl\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:42.581139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.581107 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:42.725866 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:42.725784 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:43.044838 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:43.044795 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:43.047697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:43.047667 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:43.278611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:43.278578 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:43.326852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:43.326750 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:43.565299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:43.565268 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc"] Oct 31 01:54:43.566556 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:54:43.566534 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b241c23_8b2b_4aa7_879a_ad1d8fde83f9.slice/crio-89043190f60031d71f64597247a17460b38d889c020318dd116f4849d848fef8 WatchSource:0}: Error finding container 89043190f60031d71f64597247a17460b38d889c020318dd116f4849d848fef8: Status 404 returned error can't find the container with id 89043190f60031d71f64597247a17460b38d889c020318dd116f4849d848fef8 Oct 31 01:54:44.293777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:44.286502 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:44.321633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:44.321600 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" event={"ID":"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9","Type":"ContainerStarted","Data":"30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37"} Oct 31 01:54:44.321848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:44.321676 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" event={"ID":"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9","Type":"ContainerStarted","Data":"89043190f60031d71f64597247a17460b38d889c020318dd116f4849d848fef8"} Oct 31 01:54:44.321927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:44.321862 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:44.343814 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:44.343749 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" podStartSLOduration=1.816602858 podStartE2EDuration="2.343710482s" totalImagesPullingTime="527.107624ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:54:44.339357224 +0000 UTC m=+2084.719602463" watchObservedRunningTime="2026-10-31 01:54:44.343710482 +0000 UTC m=+2084.723955706" Oct 31 01:54:44.951059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:44.951031 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:45.543172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:45.543141 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:45.680351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:45.680317 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:45.860280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:45.860190 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:46.023082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:46.023054 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:46.476502 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:46.465341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:47.150206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:47.150178 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:47.931036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:47.931001 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:48.197851 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:48.197780 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:48.332746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:48.332681 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:48.570765 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:48.570703 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:48.715432 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:48.715403 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:49.344938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:49.344825 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:50.207047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:50.199926 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:50.333653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:50.333621 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:54:50.742611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:50.742579 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:50.892030 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:50.891980 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:51.527012 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:51.525990 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:52.279626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:52.279587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:53.266925 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:53.266892 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:55.703853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:55.703794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:56.556751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:56.555870 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:57.505188 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:57.505155 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:58.437125 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:58.437077 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:59.161040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:59.161002 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:54:59.899341 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:54:59.899310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:00.037456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.037422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:00.186953 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.186876 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:55:00.192320 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.192294 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:55:00.195215 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.195196 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 01:55:00.200699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.200681 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 01:55:00.594225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.594174 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:00.769770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.769702 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:00.924570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:00.924476 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:01.592281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:01.592250 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:01.731213 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:01.731180 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:02.345199 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:02.345161 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:03.350011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:03.348541 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:04.100428 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:04.100390 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:04.261046 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:04.261017 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:04.406936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:04.406855 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:04.790276 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:55:04.790239 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:55:04.890668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:04.890629 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:05.546281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:05.546246 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:05.691033 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:05.690998 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:05.843467 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:05.843349 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:06.441381 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:06.441310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:07.157859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:07.157811 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:07.947879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:07.947848 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:08.641326 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:08.641289 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:09.251313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.251277 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb"] Oct 31 01:55:09.258053 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.258005 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.265194 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.264114 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb"] Oct 31 01:55:09.265194 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.264419 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-ed4dce-serving-cert\"" Oct 31 01:55:09.314963 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.314936 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.315102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.314974 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmbpn\" (UniqueName: \"kubernetes.io/projected/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-kube-api-access-tmbpn\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.315102 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.315084 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.315206 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.315185 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.345877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.345829 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:09.416394 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.416338 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.416588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.416413 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmbpn\" (UniqueName: \"kubernetes.io/projected/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-kube-api-access-tmbpn\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.416588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.416516 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.416588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.416584 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.418066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.418035 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.423528 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.423492 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.428275 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.428244 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.430990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.430960 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmbpn\" (UniqueName: \"kubernetes.io/projected/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-kube-api-access-tmbpn\") pod \"maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.493759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.493705 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:09.581940 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.581855 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:09.641972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.641629 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:09.903789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:09.902108 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb"] Oct 31 01:55:09.921902 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:55:09.920654 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2b427a_d7ed_4b63_875e_d56ec348ccaf.slice/crio-35dc6cc5fefd7183dacd5c0e26c6ec2b95d71d0fe74f3b9ef360e642ca588467 WatchSource:0}: Error finding container 35dc6cc5fefd7183dacd5c0e26c6ec2b95d71d0fe74f3b9ef360e642ca588467: Status 404 returned error can't find the container with id 35dc6cc5fefd7183dacd5c0e26c6ec2b95d71d0fe74f3b9ef360e642ca588467 Oct 31 01:55:10.151410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:10.150659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:10.724162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:10.724074 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" event={"ID":"9a2b427a-d7ed-4b63-875e-d56ec348ccaf","Type":"ContainerStarted","Data":"cd498b8c14ae918279829abf12128737eabd78c27e57a11cd12f437d2719c31c"} Oct 31 01:55:10.724162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:10.724133 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" event={"ID":"9a2b427a-d7ed-4b63-875e-d56ec348ccaf","Type":"ContainerStarted","Data":"35dc6cc5fefd7183dacd5c0e26c6ec2b95d71d0fe74f3b9ef360e642ca588467"} Oct 31 01:55:10.724162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:10.724157 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:10.746672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:10.746614 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" podStartSLOduration=1.335131081 podStartE2EDuration="1.746477487s" totalImagesPullingTime="411.346406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:10.745204754 +0000 UTC m=+2111.125449978" watchObservedRunningTime="2026-10-31 01:55:10.746477487 +0000 UTC m=+2111.126722710" Oct 31 01:55:10.772435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:10.772411 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:11.621588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:11.621552 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:12.248586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:12.248553 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:12.847762 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:12.846379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:13.552542 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:13.552505 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:13.783891 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:13.783856 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:14.092908 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.092822 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:14.318541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.318498 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6"] Oct 31 01:55:14.318854 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.318824 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" podUID="d10ebf99-38fa-43f9-8ccf-d1968397d926" containerName="maas-api" containerID="cri-o://697e50129e1d26429443129f0cdfb3894767f186224e3968e9764b6e2e799c23" gracePeriod=30 Oct 31 01:55:14.398626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.398592 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:14.601856 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.595369 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:14.763157 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.763109 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:14.768563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.768348 5090 generic.go:356] "Generic (PLEG): container finished" podID="d10ebf99-38fa-43f9-8ccf-d1968397d926" containerID="697e50129e1d26429443129f0cdfb3894767f186224e3968e9764b6e2e799c23" exitCode=0 Oct 31 01:55:14.768563 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.768489 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" event={"ID":"d10ebf99-38fa-43f9-8ccf-d1968397d926","Type":"ContainerDied","Data":"697e50129e1d26429443129f0cdfb3894767f186224e3968e9764b6e2e799c23"} Oct 31 01:55:14.865362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.865336 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:55:14.915263 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.915226 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d10ebf99-38fa-43f9-8ccf-d1968397d926-kube-api-access-qr6nq\" (UniqueName: \"kubernetes.io/projected/d10ebf99-38fa-43f9-8ccf-d1968397d926-kube-api-access-qr6nq\") pod \"d10ebf99-38fa-43f9-8ccf-d1968397d926\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " Oct 31 01:55:14.915985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.915956 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-maas-api-tls\") pod \"d10ebf99-38fa-43f9-8ccf-d1968397d926\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " Oct 31 01:55:14.916170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.916155 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-metrics-certs\") pod \"d10ebf99-38fa-43f9-8ccf-d1968397d926\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " Oct 31 01:55:14.916337 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.916318 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d10ebf99-38fa-43f9-8ccf-d1968397d926-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ebf99-38fa-43f9-8ccf-d1968397d926-openshift-service-ca\") pod \"d10ebf99-38fa-43f9-8ccf-d1968397d926\" (UID: \"d10ebf99-38fa-43f9-8ccf-d1968397d926\") " Oct 31 01:55:14.924975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.924922 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-maas-api-tls" pod "d10ebf99-38fa-43f9-8ccf-d1968397d926" (UID: "d10ebf99-38fa-43f9-8ccf-d1968397d926"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:55:14.924975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.924962 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d10ebf99-38fa-43f9-8ccf-d1968397d926-kube-api-access-qr6nq" pod "d10ebf99-38fa-43f9-8ccf-d1968397d926" (UID: "d10ebf99-38fa-43f9-8ccf-d1968397d926"). InnerVolumeSpecName "kube-api-access-qr6nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:55:14.925237 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.925215 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-metrics-certs" pod "d10ebf99-38fa-43f9-8ccf-d1968397d926" (UID: "d10ebf99-38fa-43f9-8ccf-d1968397d926"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:55:14.931419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.931150 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d10ebf99-38fa-43f9-8ccf-d1968397d926-openshift-service-ca" pod "d10ebf99-38fa-43f9-8ccf-d1968397d926" (UID: "d10ebf99-38fa-43f9-8ccf-d1968397d926"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:55:14.931419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.931368 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr6nq\" (UniqueName: \"kubernetes.io/projected/d10ebf99-38fa-43f9-8ccf-d1968397d926-kube-api-access-qr6nq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:14.931419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.931384 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:14.931419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.931393 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d10ebf99-38fa-43f9-8ccf-d1968397d926-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:14.931419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.931401 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d10ebf99-38fa-43f9-8ccf-d1968397d926-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:14.941665 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:14.934309 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:15.293167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:15.293128 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:15.825741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:15.821018 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" event={"ID":"d10ebf99-38fa-43f9-8ccf-d1968397d926","Type":"ContainerDied","Data":"dae9a5c08469b4500baa2911afdce44a26ab1bad73e65dc272dcb96fb435412d"} Oct 31 01:55:15.825741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:15.821087 5090 scope.go:122] "RemoveContainer" containerID="697e50129e1d26429443129f0cdfb3894767f186224e3968e9764b6e2e799c23" Oct 31 01:55:15.825741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:15.821278 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6" Oct 31 01:55:15.873870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:15.873826 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6"] Oct 31 01:55:15.879671 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:15.879624 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb8ae6-58bd4b8b77-5bsv6"] Oct 31 01:55:15.885635 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:15.885609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:16.788952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:16.788921 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 01:55:17.790741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:17.782692 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d10ebf99-38fa-43f9-8ccf-d1968397d926" path="/var/lib/kubelet/pods/d10ebf99-38fa-43f9-8ccf-d1968397d926/volumes" Oct 31 01:55:18.297449 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:18.297415 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 01:55:18.297775 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:18.297743 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 01:55:18.632362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:18.632274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:19.253743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:19.248505 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:19.760481 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:19.760445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:20.156514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:20.156437 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:20.987483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:20.986605 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:21.772452 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:21.772427 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:22.464144 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:22.464106 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:23.069010 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:23.068966 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:23.220952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:23.220922 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:23.408642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:23.408531 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:23.565731 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:23.565636 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:23.710765 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:23.710668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:23.957347 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:23.957291 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:24.103491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:24.103457 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:24.303248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:24.303211 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:25.066234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:25.063304 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:25.218846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:25.218813 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:25.453155 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:25.453085 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:25.795323 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:25.795289 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:26.300681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:26.300646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:26.468738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:26.464598 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:27.229427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:27.229397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:27.931880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:27.931847 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:28.709387 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:28.709346 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:29.368903 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:29.368864 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:29.533219 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:29.533177 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:29.690343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:29.690247 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:29.847426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:29.847395 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:30.303797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.303700 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:30.509877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.509837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:30.530317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.530268 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v"] Oct 31 01:55:30.531751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.531726 5090 cpu_manager.go:414] "containerMap: removing container" podUID="d10ebf99-38fa-43f9-8ccf-d1968397d926" containerName="maas-api" Oct 31 01:55:30.531995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.531976 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d10ebf99-38fa-43f9-8ccf-d1968397d926" containerName="maas-api" Oct 31 01:55:30.532387 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.532353 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="d10ebf99-38fa-43f9-8ccf-d1968397d926" containerName="maas-api" Oct 31 01:55:30.615480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.615398 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.622790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.621613 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-46a67e-serving-cert\"" Oct 31 01:55:30.629513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.629093 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v"] Oct 31 01:55:30.667235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.667200 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:30.786859 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.786804 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.787052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.786876 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j5kb\" (UniqueName: \"kubernetes.io/projected/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-kube-api-access-5j5kb\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.787052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.786987 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.787052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.787040 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.811876 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.811834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:30.897515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.897409 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.897515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.897492 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.897831 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.897769 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.897831 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.897802 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j5kb\" (UniqueName: \"kubernetes.io/projected/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-kube-api-access-5j5kb\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.899596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.899567 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.903808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.903783 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.904436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.904402 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.923284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.923258 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j5kb\" (UniqueName: \"kubernetes.io/projected/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-kube-api-access-5j5kb\") pod \"maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.948349 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.948321 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:30.979058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:30.979029 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:31.309778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:31.297428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:31.349821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:31.349790 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v"] Oct 31 01:55:31.350379 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:55:31.350353 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3b3d41_864b_490f_a5e1_b2ca1cff3b1b.slice/crio-cfad7c2e5ae18b8dfe79efb05d4a43f6b799b9064d1420a9f90a05f97efa5b2c WatchSource:0}: Error finding container cfad7c2e5ae18b8dfe79efb05d4a43f6b799b9064d1420a9f90a05f97efa5b2c: Status 404 returned error can't find the container with id cfad7c2e5ae18b8dfe79efb05d4a43f6b799b9064d1420a9f90a05f97efa5b2c Oct 31 01:55:31.844740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:31.837116 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:32.332313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:32.332266 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" event={"ID":"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b","Type":"ContainerStarted","Data":"c8d2a3967a903babdbe5a15b4e7d8f6d2b0b4bf9fc6769c4d8d1449075b503d6"} Oct 31 01:55:32.332526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:32.332367 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" event={"ID":"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b","Type":"ContainerStarted","Data":"cfad7c2e5ae18b8dfe79efb05d4a43f6b799b9064d1420a9f90a05f97efa5b2c"} Oct 31 01:55:32.332526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:32.332391 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:32.357424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:32.357375 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" podStartSLOduration=1.926344454 podStartE2EDuration="2.357361461s" totalImagesPullingTime="431.017007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:32.355682494 +0000 UTC m=+2132.735927746" watchObservedRunningTime="2026-10-31 01:55:32.357361461 +0000 UTC m=+2132.737606683" Oct 31 01:55:32.451861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:32.451830 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:33.403265 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:33.401988 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:34.138892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:34.138856 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:34.282651 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:34.282621 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:34.426477 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:34.426399 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:34.789926 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:55:34.789889 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:55:35.036789 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:35.036708 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:35.482227 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:35.482192 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:35.653793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:35.653758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:35.886883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:35.886786 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:36.576096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.576057 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c"] Oct 31 01:55:36.581934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.581904 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.598846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.598813 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c"] Oct 31 01:55:36.608174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.608146 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:36.619360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.619332 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.619501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.619380 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a59d732-5ada-47c4-bc33-959413d7bd71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.619741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.619698 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.619943 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.619917 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rwcq\" (UniqueName: \"kubernetes.io/projected/1a59d732-5ada-47c4-bc33-959413d7bd71-kube-api-access-4rwcq\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.633998 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.633976 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-7a850e-serving-cert\"" Oct 31 01:55:36.721070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.721038 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.721218 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.721079 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a59d732-5ada-47c4-bc33-959413d7bd71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.721274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.721255 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.721316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.721276 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rwcq\" (UniqueName: \"kubernetes.io/projected/1a59d732-5ada-47c4-bc33-959413d7bd71-kube-api-access-4rwcq\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.722070 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.722044 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a59d732-5ada-47c4-bc33-959413d7bd71-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.723795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.723771 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.723898 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.723792 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.733100 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.733071 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rwcq\" (UniqueName: \"kubernetes.io/projected/1a59d732-5ada-47c4-bc33-959413d7bd71-kube-api-access-4rwcq\") pod \"maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.758378 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.758354 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:36.914829 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.914076 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:36.932134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:36.932102 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:37.436644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:37.436514 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c"] Oct 31 01:55:37.455553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:37.455527 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:37.623985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:37.623949 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:37.902739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:37.901985 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:38.399280 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:38.399251 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 01:55:38.502014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:38.501977 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" event={"ID":"1a59d732-5ada-47c4-bc33-959413d7bd71","Type":"ContainerStarted","Data":"8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79"} Oct 31 01:55:38.519549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:38.519507 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" event={"ID":"1a59d732-5ada-47c4-bc33-959413d7bd71","Type":"ContainerStarted","Data":"f443a0956128be457dfd1e21a800ccc0e92ff38138ca12a0409ace93425b0e8d"} Oct 31 01:55:38.519864 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:38.519849 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:38.808252 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:38.608031 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" podStartSLOduration=1.9754663350000001 podStartE2EDuration="2.608007509s" totalImagesPullingTime="632.541174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:38.579405089 +0000 UTC m=+2138.959650313" watchObservedRunningTime="2026-10-31 01:55:38.608007509 +0000 UTC m=+2138.988252733" Oct 31 01:55:39.103798 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:39.100796 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:40.986000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:40.985943 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:42.173610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:42.173577 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:42.463837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:42.463687 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:42.622606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:42.622576 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:43.116307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.116270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:43.325772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.325739 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v"] Oct 31 01:55:43.326219 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.325974 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" containerID="cri-o://ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa" gracePeriod=30 Oct 31 01:55:43.796808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.796768 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:43.974749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.965567 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:43.974749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.965680 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f"] Oct 31 01:55:43.974749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.972596 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f"] Oct 31 01:55:43.974749 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:43.972708 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.063480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.063384 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6eb277-79de-4620-8243-4f41396ce223-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.064110 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.064079 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.064407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.064388 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.064618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.064600 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p54v\" (UniqueName: \"kubernetes.io/projected/eb6eb277-79de-4620-8243-4f41396ce223-kube-api-access-2p54v\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.065995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.065974 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2e6bfd-serving-cert\"" Oct 31 01:55:44.122764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.117554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:44.188134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.170665 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6eb277-79de-4620-8243-4f41396ce223-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.188601 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.188564 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.188841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.188819 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.189028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.189009 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p54v\" (UniqueName: \"kubernetes.io/projected/eb6eb277-79de-4620-8243-4f41396ce223-kube-api-access-2p54v\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.190466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.190389 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6eb277-79de-4620-8243-4f41396ce223-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.190733 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:55:44.190694 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-serving-cert: secret "maas-api-e2e-shared-b-w2-2e6bfd-serving-cert" not found Oct 31 01:55:44.190948 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:55:44.190928 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls podName:eb6eb277-79de-4620-8243-4f41396ce223 nodeName:}" failed. No retries permitted until 2026-10-31 01:55:44.690902519 +0000 UTC m=+2145.071147773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls") pod "maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" (UID: "eb6eb277-79de-4620-8243-4f41396ce223") : secret "maas-api-e2e-shared-b-w2-2e6bfd-serving-cert" not found Oct 31 01:55:44.197575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.197541 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.202775 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.202754 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p54v\" (UniqueName: \"kubernetes.io/projected/eb6eb277-79de-4620-8243-4f41396ce223-kube-api-access-2p54v\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.267539 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.267510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:44.413806 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.413660 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:44.661769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.661706 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:44.696419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.696325 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.699270 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.699243 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:44.939400 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.939363 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:44.969941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:44.969859 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:45.111100 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.111005 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:45.327695 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.327663 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:45.341790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.334755 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f"] Oct 31 01:55:45.605237 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.604984 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc"] Oct 31 01:55:45.607170 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.605238 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" podUID="5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" containerName="maas-api" containerID="cri-o://30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37" gracePeriod=30 Oct 31 01:55:45.650761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.648828 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 01:55:45.684274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.684142 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" event={"ID":"eb6eb277-79de-4620-8243-4f41396ce223","Type":"ContainerStarted","Data":"60aa4159b4395a02e733f4ac0f1c274fcd993c720d26e85d5aea397a0c9f845b"} Oct 31 01:55:45.689035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.688964 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:45.880198 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:45.878198 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:46.044207 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.042450 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:46.201856 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.201821 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:55:46.236753 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.232095 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:46.301020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.300979 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-metrics-certs\") pod \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " Oct 31 01:55:46.301184 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.301036 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls\") pod \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " Oct 31 01:55:46.301259 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.301216 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-kube-api-access-ggxvl\" (UniqueName: \"kubernetes.io/projected/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-kube-api-access-ggxvl\") pod \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " Oct 31 01:55:46.301850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.301829 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-openshift-service-ca\") pod \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\" (UID: \"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9\") " Oct 31 01:55:46.302355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.302326 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-openshift-service-ca" pod "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" (UID: "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:55:46.308376 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.308294 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-kube-api-access-ggxvl" pod "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" (UID: "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9"). InnerVolumeSpecName "kube-api-access-ggxvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:55:46.312352 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.310582 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-metrics-certs" pod "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" (UID: "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:55:46.316503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.316475 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls" pod "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" (UID: "5b241c23-8b2b-4aa7-879a-ad1d8fde83f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:55:46.403761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.403658 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:46.403761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.403696 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:46.403761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.403707 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:46.403761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.403736 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxvl\" (UniqueName: \"kubernetes.io/projected/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9-kube-api-access-ggxvl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:46.629934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.629896 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:46.719792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.715525 5090 generic.go:356] "Generic (PLEG): container finished" podID="5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" containerID="30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37" exitCode=0 Oct 31 01:55:46.719792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.715733 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" event={"ID":"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9","Type":"ContainerDied","Data":"30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37"} Oct 31 01:55:46.719792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.715822 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" event={"ID":"5b241c23-8b2b-4aa7-879a-ad1d8fde83f9","Type":"ContainerDied","Data":"89043190f60031d71f64597247a17460b38d889c020318dd116f4849d848fef8"} Oct 31 01:55:46.719792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.715852 5090 scope.go:122] "RemoveContainer" containerID="30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37" Oct 31 01:55:46.719792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.716008 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc" Oct 31 01:55:46.728269 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.728232 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" event={"ID":"eb6eb277-79de-4620-8243-4f41396ce223","Type":"ContainerStarted","Data":"ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7"} Oct 31 01:55:46.735741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.732295 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:46.757959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.757933 5090 scope.go:122] "RemoveContainer" containerID="30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37" Oct 31 01:55:46.758302 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:55:46.758281 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37\": container with ID starting with 30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37 not found: ID does not exist" containerID="30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37" Oct 31 01:55:46.758443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.758317 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37"} err="failed to get container status \"30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37\": rpc error: code = NotFound desc = could not find container \"30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37\": container with ID starting with 30b5848e5a510a134d47886003cbb1af54873e6334aa0b718cd76ce2602fae37 not found: ID does not exist" Oct 31 01:55:46.761938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.761892 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" podStartSLOduration=3.065986138 podStartE2EDuration="3.761876799s" totalImagesPullingTime="695.890661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:46.758960215 +0000 UTC m=+2147.139205452" watchObservedRunningTime="2026-10-31 01:55:46.761876799 +0000 UTC m=+2147.142122022" Oct 31 01:55:46.787475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.787407 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc"] Oct 31 01:55:46.791285 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:46.791255 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dbff33-67fc87cf89-7jklc"] Oct 31 01:55:47.143871 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:47.143296 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:47.749453 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:47.749423 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" path="/var/lib/kubelet/pods/5b241c23-8b2b-4aa7-879a-ad1d8fde83f9/volumes" Oct 31 01:55:47.911889 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:47.911807 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:48.665737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.663993 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:55:48.734927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.734896 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-kube-api-access-xzvsm\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.734955 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-socket\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.734976 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-data\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735030 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-certs\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735090 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3cbc25-a030-4045-8329-58df5508742e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3cbc25-a030-4045-8329-58df5508742e-istiod-ca-cert\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735137 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-credential-socket\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735256 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-envoy\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735295 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-istio-token\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-istio-token\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735313 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-socket" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:55:48.735558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735357 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-data" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:55:48.735558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735365 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c3cbc25-a030-4045-8329-58df5508742e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3cbc25-a030-4045-8329-58df5508742e-istio-podinfo\") pod \"5c3cbc25-a030-4045-8329-58df5508742e\" (UID: \"5c3cbc25-a030-4045-8329-58df5508742e\") " Oct 31 01:55:48.735558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735386 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-certs" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:55:48.735558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735530 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-credential-socket" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:55:48.735558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.735537 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3cbc25-a030-4045-8329-58df5508742e-istiod-ca-cert" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:55:48.736139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.736010 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.736139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.736033 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.736139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.736048 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.736139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.736062 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c3cbc25-a030-4045-8329-58df5508742e-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.736139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.736074 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.739509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.738839 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-envoy" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:55:48.739509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.738850 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-istio-token" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:55:48.739509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.738860 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c3cbc25-a030-4045-8329-58df5508742e-istio-podinfo" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 01:55:48.739509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.738872 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-kube-api-access-xzvsm" pod "5c3cbc25-a030-4045-8329-58df5508742e" (UID: "5c3cbc25-a030-4045-8329-58df5508742e"). InnerVolumeSpecName "kube-api-access-xzvsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:55:48.745241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.745220 5090 generic.go:356] "Generic (PLEG): container finished" podID="5c3cbc25-a030-4045-8329-58df5508742e" containerID="ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa" exitCode=0 Oct 31 01:55:48.745364 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.745282 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" Oct 31 01:55:48.745364 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.745282 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" event={"ID":"5c3cbc25-a030-4045-8329-58df5508742e","Type":"ContainerDied","Data":"ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa"} Oct 31 01:55:48.745473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.745421 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v" event={"ID":"5c3cbc25-a030-4045-8329-58df5508742e","Type":"ContainerDied","Data":"9c94be87c95a36bf66a2f87c4e36ffbbbf747564e1fa67de4e9d195ff4eb6d67"} Oct 31 01:55:48.745473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.745460 5090 scope.go:122] "RemoveContainer" containerID="ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa" Oct 31 01:55:48.764919 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.763284 5090 scope.go:122] "RemoveContainer" containerID="ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa" Oct 31 01:55:48.764919 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:55:48.763631 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa\": container with ID starting with ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa not found: ID does not exist" containerID="ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa" Oct 31 01:55:48.764919 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.763668 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa"} err="failed to get container status \"ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa\": rpc error: code = NotFound desc = could not find container \"ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa\": container with ID starting with ceafd2c567d16554468b122cbbefc476890feebec98458946ab5b0a3ee9aabaa not found: ID does not exist" Oct 31 01:55:48.781236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.781207 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v"] Oct 31 01:55:48.785226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.785196 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-ckw8v"] Oct 31 01:55:48.837391 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.837357 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c3cbc25-a030-4045-8329-58df5508742e-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.838003 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.837978 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.838131 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.838005 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c3cbc25-a030-4045-8329-58df5508742e-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.838131 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.838030 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzvsm\" (UniqueName: \"kubernetes.io/projected/5c3cbc25-a030-4045-8329-58df5508742e-kube-api-access-xzvsm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:55:48.939995 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:48.939909 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:49.113115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:49.113014 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:49.630943 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:49.630915 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:49.752281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:49.752251 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3cbc25-a030-4045-8329-58df5508742e" path="/var/lib/kubelet/pods/5c3cbc25-a030-4045-8329-58df5508742e/volumes" Oct 31 01:55:49.893000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:49.892919 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:50.112399 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.112360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:50.169570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.169492 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv"] Oct 31 01:55:50.170201 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.170184 5090 cpu_manager.go:414] "containerMap: removing container" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" Oct 31 01:55:50.170290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.170203 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" Oct 31 01:55:50.170290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.170230 5090 cpu_manager.go:414] "containerMap: removing container" podUID="5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" containerName="maas-api" Oct 31 01:55:50.170290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.170235 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" containerName="maas-api" Oct 31 01:55:50.170386 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.170328 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3cbc25-a030-4045-8329-58df5508742e" containerName="istio-proxy" Oct 31 01:55:50.170386 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.170339 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b241c23-8b2b-4aa7-879a-ad1d8fde83f9" containerName="maas-api" Oct 31 01:55:50.175640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.175620 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.181777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.181747 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-6c54a1\"" Oct 31 01:55:50.182224 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.182204 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-6c54a1-dockercfg-mpjdf\"" Oct 31 01:55:50.187486 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.187461 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv"] Oct 31 01:55:50.227380 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.227351 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9"] Oct 31 01:55:50.231503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.231486 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.240567 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.240541 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9"] Oct 31 01:55:50.268355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.263989 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:50.358741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.358243 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4333d31c-1c31-4394-a273-3fabbc1751ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.358741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.358431 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rbtq\" (UniqueName: \"kubernetes.io/projected/4333d31c-1c31-4394-a273-3fabbc1751ca-kube-api-access-9rbtq\") pod \"payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.461579 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.461494 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbtq\" (UniqueName: \"kubernetes.io/projected/4333d31c-1c31-4394-a273-3fabbc1751ca-kube-api-access-9rbtq\") pod \"payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.461840 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.461819 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c26e-106b-4d86-9885-9026a336feca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.462181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.462158 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtqd\" (UniqueName: \"kubernetes.io/projected/7bd7c26e-106b-4d86-9885-9026a336feca-kube-api-access-bqtqd\") pod \"payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.462372 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.462356 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4333d31c-1c31-4394-a273-3fabbc1751ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.463554 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.463359 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4333d31c-1c31-4394-a273-3fabbc1751ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.489264 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.485445 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbtq\" (UniqueName: \"kubernetes.io/projected/4333d31c-1c31-4394-a273-3fabbc1751ca-kube-api-access-9rbtq\") pod \"payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.493885 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.493156 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:50.564171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.564126 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c26e-106b-4d86-9885-9026a336feca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.564390 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.564289 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtqd\" (UniqueName: \"kubernetes.io/projected/7bd7c26e-106b-4d86-9885-9026a336feca-kube-api-access-bqtqd\") pod \"payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.571178 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.571100 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c26e-106b-4d86-9885-9026a336feca-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.654519 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.635040 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtqd\" (UniqueName: \"kubernetes.io/projected/7bd7c26e-106b-4d86-9885-9026a336feca-kube-api-access-bqtqd\") pod \"payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.770830 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.758397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:50.871192 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.871148 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:50.939873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:50.939843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:51.070794 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.069107 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv"] Oct 31 01:55:51.080855 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:55:51.080833 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4333d31c_1c31_4394_a273_3fabbc1751ca.slice/crio-d1f850793aedee9338a1e845e7c3439bcfd573940f86357f4ff95891a17bb663 WatchSource:0}: Error finding container d1f850793aedee9338a1e845e7c3439bcfd573940f86357f4ff95891a17bb663: Status 404 returned error can't find the container with id d1f850793aedee9338a1e845e7c3439bcfd573940f86357f4ff95891a17bb663 Oct 31 01:55:51.088186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.088165 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:51.252739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.252683 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:51.294117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.294087 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9"] Oct 31 01:55:51.296946 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:55:51.296909 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd7c26e_106b_4d86_9885_9026a336feca.slice/crio-8f298bfaaffbdee1a7cdd94f948d1cf8763eabfcea8341a918daf25c4db18934 WatchSource:0}: Error finding container 8f298bfaaffbdee1a7cdd94f948d1cf8763eabfcea8341a918daf25c4db18934: Status 404 returned error can't find the container with id 8f298bfaaffbdee1a7cdd94f948d1cf8763eabfcea8341a918daf25c4db18934 Oct 31 01:55:51.945584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.945545 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" event={"ID":"7bd7c26e-106b-4d86-9885-9026a336feca","Type":"ContainerStarted","Data":"f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0"} Oct 31 01:55:51.946115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.945595 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" event={"ID":"7bd7c26e-106b-4d86-9885-9026a336feca","Type":"ContainerStarted","Data":"8f298bfaaffbdee1a7cdd94f948d1cf8763eabfcea8341a918daf25c4db18934"} Oct 31 01:55:51.946115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.945654 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:55:51.947591 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.947566 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" event={"ID":"4333d31c-1c31-4394-a273-3fabbc1751ca","Type":"ContainerStarted","Data":"7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9"} Oct 31 01:55:51.947729 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.947601 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" event={"ID":"4333d31c-1c31-4394-a273-3fabbc1751ca","Type":"ContainerStarted","Data":"d1f850793aedee9338a1e845e7c3439bcfd573940f86357f4ff95891a17bb663"} Oct 31 01:55:51.947729 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.947622 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:55:51.963946 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.963901 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" podStartSLOduration=1.96388455 podStartE2EDuration="1.96388455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:51.961673619 +0000 UTC m=+2152.341918842" watchObservedRunningTime="2026-10-31 01:55:51.96388455 +0000 UTC m=+2152.344129774" Oct 31 01:55:51.982247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:51.982200 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" podStartSLOduration=1.9821842219999999 podStartE2EDuration="1.982184222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:51.977540448 +0000 UTC m=+2152.357785674" watchObservedRunningTime="2026-10-31 01:55:51.982184222 +0000 UTC m=+2152.362429446" Oct 31 01:55:52.052594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.052559 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg"] Oct 31 01:55:52.065080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.065052 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg"] Oct 31 01:55:52.065243 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.065171 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.078757 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.073995 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-6d7f5d-serving-cert\"" Oct 31 01:55:52.139504 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.139477 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:52.216021 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.215933 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.216183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.216101 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46cdd57-3bb7-4b44-863c-73c12433a40d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.216183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.216124 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.216183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.216171 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjr7\" (UniqueName: \"kubernetes.io/projected/d46cdd57-3bb7-4b44-863c-73c12433a40d-kube-api-access-bwjr7\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.317440 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.317379 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46cdd57-3bb7-4b44-863c-73c12433a40d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.317440 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.317436 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.317756 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.317505 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjr7\" (UniqueName: \"kubernetes.io/projected/d46cdd57-3bb7-4b44-863c-73c12433a40d-kube-api-access-bwjr7\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.317756 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.317630 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.319022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.318994 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46cdd57-3bb7-4b44-863c-73c12433a40d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.321056 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.321030 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.322489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.322466 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.330440 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.330414 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjr7\" (UniqueName: \"kubernetes.io/projected/d46cdd57-3bb7-4b44-863c-73c12433a40d-kube-api-access-bwjr7\") pod \"maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.340545 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.340515 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:52.398040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.397993 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:52.716485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.716397 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg"] Oct 31 01:55:52.725756 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:55:52.725687 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46cdd57_3bb7_4b44_863c_73c12433a40d.slice/crio-43a3237b8eccf608d8d4ed52aab45c4648372a7a71ab83a8591ec4e5fd7d6d44 WatchSource:0}: Error finding container 43a3237b8eccf608d8d4ed52aab45c4648372a7a71ab83a8591ec4e5fd7d6d44: Status 404 returned error can't find the container with id 43a3237b8eccf608d8d4ed52aab45c4648372a7a71ab83a8591ec4e5fd7d6d44 Oct 31 01:55:52.954870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:52.954833 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" event={"ID":"d46cdd57-3bb7-4b44-863c-73c12433a40d","Type":"ContainerStarted","Data":"43a3237b8eccf608d8d4ed52aab45c4648372a7a71ab83a8591ec4e5fd7d6d44"} Oct 31 01:55:53.093547 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:53.093513 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:53.755253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:53.755214 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 01:55:53.756314 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:53.756279 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:53.966682 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:53.966647 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" event={"ID":"d46cdd57-3bb7-4b44-863c-73c12433a40d","Type":"ContainerStarted","Data":"9ae6dd245597836a1bf18a552bd9291a9172d3e64802797595174c1f95cadb91"} Oct 31 01:55:53.967089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:53.966708 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:55:54.003287 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:54.003228 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" podStartSLOduration=1.587286875 podStartE2EDuration="2.003212948s" totalImagesPullingTime="415.926073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:53.999012186 +0000 UTC m=+2154.379257409" watchObservedRunningTime="2026-10-31 01:55:54.003212948 +0000 UTC m=+2154.383458171" Oct 31 01:55:54.055564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:54.055497 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:54.208509 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:54.208442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:54.620587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:54.620553 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:54.785441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:54.785411 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:54.982008 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:54.981927 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:55.357583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.357555 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:55.425011 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.424973 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq"] Oct 31 01:55:55.430455 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.430424 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:55.439867 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.439842 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq"] Oct 31 01:55:55.510430 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.510390 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ae5a1c-dockercfg-sshzj\"" Oct 31 01:55:55.510628 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.510485 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ae5a1c\"" Oct 31 01:55:55.603793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.600100 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:55.604104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.604072 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbcnk\" (UniqueName: \"kubernetes.io/projected/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-kube-api-access-nbcnk\") pod \"payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:55.705739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.705623 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:55.706005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.705977 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbcnk\" (UniqueName: \"kubernetes.io/projected/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-kube-api-access-nbcnk\") pod \"payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:55.706398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.706380 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:55.724068 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.724039 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbcnk\" (UniqueName: \"kubernetes.io/projected/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-kube-api-access-nbcnk\") pod \"payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:55.747438 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:55.747408 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:56.038489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:56.038422 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq"] Oct 31 01:55:56.040385 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:55:56.040354 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf2337f_bdd6_4ed5_ad43_83c528eb8cb7.slice/crio-f12094e36f0b1af91205e9d1cd61c594613201384ca060474f741d8e8b0c8efd WatchSource:0}: Error finding container f12094e36f0b1af91205e9d1cd61c594613201384ca060474f741d8e8b0c8efd: Status 404 returned error can't find the container with id f12094e36f0b1af91205e9d1cd61c594613201384ca060474f741d8e8b0c8efd Oct 31 01:55:56.389140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:56.389069 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:56.549562 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:56.549517 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:56.991066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:56.990636 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" event={"ID":"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7","Type":"ContainerStarted","Data":"f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9"} Oct 31 01:55:56.991066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:56.990702 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" event={"ID":"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7","Type":"ContainerStarted","Data":"f12094e36f0b1af91205e9d1cd61c594613201384ca060474f741d8e8b0c8efd"} Oct 31 01:55:56.991066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:56.990753 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:55:57.011119 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:57.011071 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" podStartSLOduration=2.011057082 podStartE2EDuration="2.011057082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:55:57.008233165 +0000 UTC m=+2157.388478388" watchObservedRunningTime="2026-10-31 01:55:57.011057082 +0000 UTC m=+2157.391302319" Oct 31 01:55:57.513738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:57.507176 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:58.341028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.340983 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz"] Oct 31 01:55:58.347359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.346548 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.367146 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.364053 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-6c54a1-serving-cert\"" Oct 31 01:55:58.377124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.368635 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz"] Oct 31 01:55:58.389505 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.389468 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.389702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.389681 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff8334-9229-4a72-89b7-5128f0d59380-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.389847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.389827 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.389921 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.389902 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-659j5\" (UniqueName: \"kubernetes.io/projected/3aff8334-9229-4a72-89b7-5128f0d59380-kube-api-access-659j5\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.441605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.441570 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:58.491644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.491606 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.493887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.493851 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-659j5\" (UniqueName: \"kubernetes.io/projected/3aff8334-9229-4a72-89b7-5128f0d59380-kube-api-access-659j5\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.494072 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.494046 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.494247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.494222 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff8334-9229-4a72-89b7-5128f0d59380-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.495144 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.495121 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff8334-9229-4a72-89b7-5128f0d59380-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.498841 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.498817 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.499274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.499253 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.507681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.507660 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-659j5\" (UniqueName: \"kubernetes.io/projected/3aff8334-9229-4a72-89b7-5128f0d59380-kube-api-access-659j5\") pod \"maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.596276 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.596156 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:58.680019 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.679985 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:55:58.833120 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.833083 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:58.989259 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:58.989227 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz"] Oct 31 01:55:58.989645 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:55:58.989615 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aff8334_9229_4a72_89b7_5128f0d59380.slice/crio-a64bb80412e40cde7c37ebb8f33e074410bb4063c6571610d0da3edb78d949cf WatchSource:0}: Error finding container a64bb80412e40cde7c37ebb8f33e074410bb4063c6571610d0da3edb78d949cf: Status 404 returned error can't find the container with id a64bb80412e40cde7c37ebb8f33e074410bb4063c6571610d0da3edb78d949cf Oct 31 01:55:59.006242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:59.006167 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" event={"ID":"3aff8334-9229-4a72-89b7-5128f0d59380","Type":"ContainerStarted","Data":"a64bb80412e40cde7c37ebb8f33e074410bb4063c6571610d0da3edb78d949cf"} Oct 31 01:55:59.422900 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:59.422664 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:59.568851 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:59.568803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:55:59.979420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:55:59.979386 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 01:56:00.020759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:00.017232 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" event={"ID":"3aff8334-9229-4a72-89b7-5128f0d59380","Type":"ContainerStarted","Data":"a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db"} Oct 31 01:56:00.020759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:00.017375 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:56:00.040185 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:00.039353 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" podStartSLOduration=1.5543375940000002 podStartE2EDuration="2.039338214s" totalImagesPullingTime="485.00062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:00.035947059 +0000 UTC m=+2160.416192301" watchObservedRunningTime="2026-10-31 01:56:00.039338214 +0000 UTC m=+2160.419583436" Oct 31 01:56:00.205626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:00.205584 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:00.720262 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:00.720184 5090 scope.go:122] "RemoveContainer" containerID="bde2e581acf9440fc1e3c3d47573f6764bfecc8ccf9661c59cc53324fc1b7ea6" Oct 31 01:56:00.974797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:00.967283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:01.293802 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.293765 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc"] Oct 31 01:56:01.301115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.301091 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.306585 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.306552 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc"] Oct 31 01:56:01.309035 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.309011 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed4dce\"/\"kube-root-ca.crt\"" Oct 31 01:56:01.309153 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.309131 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed4dce\"/\"openshift-service-ca.crt\"" Oct 31 01:56:01.309226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.309206 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed4dce\"/\"default-dockercfg-c6zrw\"" Oct 31 01:56:01.309290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.309271 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed4dce\"/\"auth-test-model-ed4dce-kserve-self-signed-certs\"" Oct 31 01:56:01.336702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.336671 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-tmp-dir\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.336904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.336727 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-home\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.336904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.336891 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8d4x\" (UniqueName: \"kubernetes.io/projected/9c95bc40-df72-46ba-9b00-a725595908ec-kube-api-access-q8d4x\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.337020 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.336992 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-model-cache\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.337122 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.337103 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c95bc40-df72-46ba-9b00-a725595908ec-tls-certs\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.337195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.337151 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-dshm\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.366638 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.366603 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:01.439082 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439034 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-tmp-dir\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439101 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-home\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439160 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8d4x\" (UniqueName: \"kubernetes.io/projected/9c95bc40-df72-46ba-9b00-a725595908ec-kube-api-access-q8d4x\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439251 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-model-cache\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439318 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c95bc40-df72-46ba-9b00-a725595908ec-tls-certs\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439360 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-dshm\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439535 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439513 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-tmp-dir\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439595 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439570 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-home\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.439851 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.439827 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-model-cache\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.441893 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.441867 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-dshm\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.456331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.456298 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c95bc40-df72-46ba-9b00-a725595908ec-tls-certs\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.485383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.482685 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8d4x\" (UniqueName: \"kubernetes.io/projected/9c95bc40-df72-46ba-9b00-a725595908ec-kube-api-access-q8d4x\") pod \"auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.621496 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.621421 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:01.851656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.851534 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:01.925334 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:01.925247 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc"] Oct 31 01:56:01.930377 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:56:01.930333 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c95bc40_df72_46ba_9b00_a725595908ec.slice/crio-2223f99e180de1e4f1ae1f14bb192f023cd7da08328f6e01e812ecfcc3a219cc WatchSource:0}: Error finding container 2223f99e180de1e4f1ae1f14bb192f023cd7da08328f6e01e812ecfcc3a219cc: Status 404 returned error can't find the container with id 2223f99e180de1e4f1ae1f14bb192f023cd7da08328f6e01e812ecfcc3a219cc Oct 31 01:56:02.034769 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.034693 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" event={"ID":"9c95bc40-df72-46ba-9b00-a725595908ec","Type":"ContainerStarted","Data":"4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8"} Oct 31 01:56:02.034917 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.034784 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" event={"ID":"9c95bc40-df72-46ba-9b00-a725595908ec","Type":"ContainerStarted","Data":"2223f99e180de1e4f1ae1f14bb192f023cd7da08328f6e01e812ecfcc3a219cc"} Oct 31 01:56:02.058042 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.057983 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" podStartSLOduration=1.057967292 podStartE2EDuration="1.057967292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:02.054690269 +0000 UTC m=+2162.434935519" watchObservedRunningTime="2026-10-31 01:56:02.057967292 +0000 UTC m=+2162.438212514" Oct 31 01:56:02.206656 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.206554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:02.622125 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.622090 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:02.730778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.730738 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:02.899469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.895856 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:02.966101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.966072 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 01:56:02.966573 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:02.966554 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 01:56:03.181600 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.181508 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:03.575960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.575913 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt"] Oct 31 01:56:03.581553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.581526 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.589019 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.588949 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ae5a1c-serving-cert\"" Oct 31 01:56:03.595533 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.595489 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt"] Oct 31 01:56:03.694879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.694840 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:03.731781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.731707 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2b839-b6a1-46fe-8678-c6d984684533-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.731972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.731851 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.731972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.731936 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.732088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.732028 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgzh\" (UniqueName: \"kubernetes.io/projected/f3b2b839-b6a1-46fe-8678-c6d984684533-kube-api-access-cwgzh\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.833804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.833666 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2b839-b6a1-46fe-8678-c6d984684533-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.833975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.833813 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.833975 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.833886 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.834667 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.834482 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgzh\" (UniqueName: \"kubernetes.io/projected/f3b2b839-b6a1-46fe-8678-c6d984684533-kube-api-access-cwgzh\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.834846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.834703 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2b839-b6a1-46fe-8678-c6d984684533-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.840501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.840443 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.840501 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.840492 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.849963 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.849932 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgzh\" (UniqueName: \"kubernetes.io/projected/f3b2b839-b6a1-46fe-8678-c6d984684533-kube-api-access-cwgzh\") pod \"maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:03.895169 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.893315 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:03.957047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:03.957019 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:04.044798 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:04.044479 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:04.262953 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:04.262917 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt"] Oct 31 01:56:04.673259 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:04.673132 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:04.790328 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:56:04.790298 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:56:05.080870 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:05.080836 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" event={"ID":"f3b2b839-b6a1-46fe-8678-c6d984684533","Type":"ContainerStarted","Data":"0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526"} Oct 31 01:56:05.081051 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:05.080890 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" event={"ID":"f3b2b839-b6a1-46fe-8678-c6d984684533","Type":"ContainerStarted","Data":"47150cba7581bbfa6d819ccb1f2c3bad3c27f7d3be56eae551bd38d677089c92"} Oct 31 01:56:05.081051 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:05.080935 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:05.104908 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:05.104864 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" podStartSLOduration=1.668943354 podStartE2EDuration="2.104850627s" totalImagesPullingTime="435.907273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:05.102458269 +0000 UTC m=+2165.482703492" watchObservedRunningTime="2026-10-31 01:56:05.104850627 +0000 UTC m=+2165.485095849" Oct 31 01:56:05.463581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:05.463487 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:06.032299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:06.032268 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 01:56:06.269817 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:06.269415 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:07.049136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:07.049099 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:07.518419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:07.518393 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:07.677319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:07.677286 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:07.894355 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:07.894286 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:07.999571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:07.999545 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 01:56:08.055880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:08.055847 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:08.371805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:08.371774 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:08.738689 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:08.738605 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:08.928792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:08.916641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:09.118738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:09.118243 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:09.741344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:09.741261 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:10.664801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:10.661976 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:11.100344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:11.100317 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 01:56:11.458103 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:11.458009 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:11.622777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:11.622746 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:11.636076 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:11.636054 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:12.154489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:12.154459 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 01:56:12.329204 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:12.329165 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:12.707522 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:12.707493 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:12.859706 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:12.859675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:13.237581 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.237539 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:13.496069 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.496037 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q"] Oct 31 01:56:13.501960 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.501938 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.527223 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.527050 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q"] Oct 31 01:56:13.545291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.545258 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.545443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.545320 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.545443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.545377 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncck2\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-kube-api-access-ncck2\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.545589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.545484 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.545652 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.545607 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.545772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.545749 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.545914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.545823 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.546139 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.546112 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.546208 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.546148 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647209 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647247 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647255 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647504 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647303 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647504 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647339 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647504 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647483 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncck2\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-kube-api-access-ncck2\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647679 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647637 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647691 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-data\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647729 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-credential-socket\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.647927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647904 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.648059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.647956 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.648059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.648009 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.648059 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.648046 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.648708 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.648525 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-certs\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.649001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.648975 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istiod-ca-cert\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.650456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.650436 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-podinfo\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.650574 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.650551 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-envoy\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.659423 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.659397 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncck2\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-kube-api-access-ncck2\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.659507 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.659457 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-token\") pod \"e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.826848 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.826764 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:13.947756 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:13.947666 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:14.086458 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.086379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:14.130242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.130209 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q"] Oct 31 01:56:14.134673 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:56:14.134628 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006e6dab_5f1e_4c66_a2b6_42d2e5152628.slice/crio-add69bd8002bc484f4dd762c30ab9e3db5fee6b360dff073998b7c01e1f1b79e WatchSource:0}: Error finding container add69bd8002bc484f4dd762c30ab9e3db5fee6b360dff073998b7c01e1f1b79e: Status 404 returned error can't find the container with id add69bd8002bc484f4dd762c30ab9e3db5fee6b360dff073998b7c01e1f1b79e Oct 31 01:56:14.137778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.137740 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:56:14.137882 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.137818 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:56:14.137882 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.137854 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:56:14.167805 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.167775 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" event={"ID":"006e6dab-5f1e-4c66-a2b6-42d2e5152628","Type":"ContainerStarted","Data":"add69bd8002bc484f4dd762c30ab9e3db5fee6b360dff073998b7c01e1f1b79e"} Oct 31 01:56:14.265680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.265646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:14.829468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:14.829435 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:15.187796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:15.182905 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" event={"ID":"006e6dab-5f1e-4c66-a2b6-42d2e5152628","Type":"ContainerStarted","Data":"28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d"} Oct 31 01:56:15.207085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:15.206974 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podStartSLOduration=2.206959181 podStartE2EDuration="2.206959181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:15.204414516 +0000 UTC m=+2175.584659800" watchObservedRunningTime="2026-10-31 01:56:15.206959181 +0000 UTC m=+2175.587204407" Oct 31 01:56:15.827584 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:15.827557 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:15.828879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:15.828703 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:56:15.828879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:15.828788 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 01:56:15.912927 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:15.912902 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:16.524136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.524102 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl"] Oct 31 01:56:16.528426 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.528406 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.542494 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.542471 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-46a67e\"/\"auth-test-model-46a67e-kserve-self-signed-certs\"" Oct 31 01:56:16.543036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.543018 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-46a67e\"/\"default-dockercfg-v7s8f\"" Oct 31 01:56:16.561253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.561231 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl"] Oct 31 01:56:16.615178 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.615005 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-46a67e\"/\"openshift-service-ca.crt\"" Oct 31 01:56:16.615178 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.615143 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-46a67e\"/\"kube-root-ca.crt\"" Oct 31 01:56:16.622950 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.622860 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:16.660186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.660150 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj9r\" (UniqueName: \"kubernetes.io/projected/01f10455-5aa7-4d95-8893-7ae79b8bfe00-kube-api-access-nsj9r\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.660357 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.660316 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tmp-dir\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.660428 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.660404 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-model-cache\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.660485 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.660470 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-dshm\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.660588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.660573 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-home\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.660643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.660599 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tls-certs\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.762737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.762290 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tmp-dir\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.762737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.762376 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-model-cache\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.762737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.762446 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-dshm\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.762737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.762489 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-home\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.762737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.762515 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tls-certs\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.762737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.762686 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj9r\" (UniqueName: \"kubernetes.io/projected/01f10455-5aa7-4d95-8893-7ae79b8bfe00-kube-api-access-nsj9r\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.781796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.763627 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tmp-dir\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.781796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.764221 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-model-cache\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.781796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.770060 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-home\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.793759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.793406 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tls-certs\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.810737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.804753 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-dshm\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.840735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.830775 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj9r\" (UniqueName: \"kubernetes.io/projected/01f10455-5aa7-4d95-8893-7ae79b8bfe00-kube-api-access-nsj9r\") pod \"auth-test-model-46a67e-kserve-64896d6dbf-t7lhl\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:16.840735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.831193 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:16.840735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.833056 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:56:16.840735 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.833103 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 01:56:16.877508 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:16.876310 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:17.272146 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:17.272110 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl"] Oct 31 01:56:17.718036 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:17.717942 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:17.828226 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:17.828191 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:56:17.828427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:17.828257 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 01:56:18.231894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.231858 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" event={"ID":"01f10455-5aa7-4d95-8893-7ae79b8bfe00","Type":"ContainerStarted","Data":"a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6"} Oct 31 01:56:18.232403 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.231908 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" event={"ID":"01f10455-5aa7-4d95-8893-7ae79b8bfe00","Type":"ContainerStarted","Data":"c0ca1305012c140fa1477d22cc7f4d59b449acedece7563189c80f2903a2b0f4"} Oct 31 01:56:18.253142 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.253093 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" podStartSLOduration=2.25308006 podStartE2EDuration="2.25308006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:18.249814759 +0000 UTC m=+2178.630060010" watchObservedRunningTime="2026-10-31 01:56:18.25308006 +0000 UTC m=+2178.633325340" Oct 31 01:56:18.653610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.652731 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5"] Oct 31 01:56:18.658508 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.658482 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:18.665604 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.665572 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c8b702-serving-cert\"" Oct 31 01:56:18.672180 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.672146 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5"] Oct 31 01:56:18.734588 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.734557 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:18.827232 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.827194 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:56:18.827510 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.827258 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 01:56:18.895755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.881858 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:18.909819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.906909 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj64d\" (UniqueName: \"kubernetes.io/projected/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-kube-api-access-zj64d\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:18.909819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.906979 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-metrics-certs\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:18.909819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.907292 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-openshift-service-ca\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:18.909819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.907537 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-maas-api-tls\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:18.909819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:18.908617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:19.009248 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.009208 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-openshift-service-ca\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.009422 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.009344 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-maas-api-tls\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.009511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.009452 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj64d\" (UniqueName: \"kubernetes.io/projected/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-kube-api-access-zj64d\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.009511 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.009481 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-metrics-certs\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.010152 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.010122 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-openshift-service-ca\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.013632 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.013600 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-metrics-certs\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.014970 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.014948 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-maas-api-tls\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.022268 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.022241 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj64d\" (UniqueName: \"kubernetes.io/projected/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-kube-api-access-zj64d\") pod \"maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.022905 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.022849 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:19.345778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.345609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:19.538099 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.538069 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5"] Oct 31 01:56:19.809738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.809396 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:19.828697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.828359 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:56:19.828697 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:19.828430 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 01:56:20.356146 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:20.356056 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" event={"ID":"fc66f2a7-2c3b-4ec9-86db-60c8276734e6","Type":"ContainerStarted","Data":"0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b"} Oct 31 01:56:20.356640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:20.356306 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:20.356640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:20.356331 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" event={"ID":"fc66f2a7-2c3b-4ec9-86db-60c8276734e6","Type":"ContainerStarted","Data":"90be8a15c4d7e4db4aaf8185c67e7abca7f64be8e23441fdc2d7c61c9544530a"} Oct 31 01:56:20.378454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:20.378404 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" podStartSLOduration=1.8238438769999998 podStartE2EDuration="2.378388508s" totalImagesPullingTime="554.544631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:20.377312712 +0000 UTC m=+2180.757557955" watchObservedRunningTime="2026-10-31 01:56:20.378388508 +0000 UTC m=+2180.758633761" Oct 31 01:56:20.728039 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:20.727939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:20.828048 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:20.828010 5090 patch_prober.go:29] interesting pod/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 31 01:56:20.828242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:20.828077 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 31 01:56:21.497353 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:21.497315 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:21.873234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:21.873141 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:21.873690 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:21.873662 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:21.874576 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:21.874556 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 01:56:22.133666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:22.133584 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:22.908411 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:22.906287 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:23.553587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:23.553550 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:24.145847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.142693 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9"] Oct 31 01:56:24.151398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.150005 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.159873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.159850 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9"] Oct 31 01:56:24.183158 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.183131 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.183299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.183191 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.183407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.183378 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aa30b9-8948-46ab-b29e-655807afa437-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.183538 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.183520 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwfl\" (UniqueName: \"kubernetes.io/projected/82aa30b9-8948-46ab-b29e-655807afa437-kube-api-access-lkwfl\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.207174 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.207147 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-e57add-serving-cert\"" Oct 31 01:56:24.287431 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.287388 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.287640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.287531 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aa30b9-8948-46ab-b29e-655807afa437-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.287751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.287639 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwfl\" (UniqueName: \"kubernetes.io/projected/82aa30b9-8948-46ab-b29e-655807afa437-kube-api-access-lkwfl\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.287880 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.287842 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.288760 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:56:24.288701 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-serving-cert: secret "maas-api-e2e-shared-b-w0-e57add-serving-cert" not found Oct 31 01:56:24.288958 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:56:24.288806 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls podName:82aa30b9-8948-46ab-b29e-655807afa437 nodeName:}" failed. No retries permitted until 2026-10-31 01:56:24.788783818 +0000 UTC m=+2185.169029019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls") pod "maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" (UID: "82aa30b9-8948-46ab-b29e-655807afa437") : secret "maas-api-e2e-shared-b-w0-e57add-serving-cert" not found Oct 31 01:56:24.290254 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.290213 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aa30b9-8948-46ab-b29e-655807afa437-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.291481 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.291457 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.316173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.316145 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwfl\" (UniqueName: \"kubernetes.io/projected/82aa30b9-8948-46ab-b29e-655807afa437-kube-api-access-lkwfl\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.333420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.333388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:24.493607 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.493578 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:24.796790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.796245 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:24.813739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:24.812445 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:25.079552 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:25.079478 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:25.295648 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:25.295617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:25.371472 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:25.371445 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9"] Oct 31 01:56:25.373986 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:56:25.373945 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82aa30b9_8948_46ab_b29e_655807afa437.slice/crio-72677fe51678e9d24227e52521b00644c88af88a897c2ee6d0fff77d4d8848c3 WatchSource:0}: Error finding container 72677fe51678e9d24227e52521b00644c88af88a897c2ee6d0fff77d4d8848c3: Status 404 returned error can't find the container with id 72677fe51678e9d24227e52521b00644c88af88a897c2ee6d0fff77d4d8848c3 Oct 31 01:56:25.395663 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:25.395632 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" event={"ID":"82aa30b9-8948-46ab-b29e-655807afa437","Type":"ContainerStarted","Data":"72677fe51678e9d24227e52521b00644c88af88a897c2ee6d0fff77d4d8848c3"} Oct 31 01:56:25.947497 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:25.947469 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:26.371195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:26.371166 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:56:26.421490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:26.421443 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" event={"ID":"82aa30b9-8948-46ab-b29e-655807afa437","Type":"ContainerStarted","Data":"29d25db6f101a6e37c2e5d1de3066ace637d246f907d0d29cd120324aa69d068"} Oct 31 01:56:26.421686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:26.421631 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:26.445796 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:26.445707 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" podStartSLOduration=2.016467771 podStartE2EDuration="2.445690095s" totalImagesPullingTime="429.222324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:26.442493874 +0000 UTC m=+2186.822739098" watchObservedRunningTime="2026-10-31 01:56:26.445690095 +0000 UTC m=+2186.825935321" Oct 31 01:56:26.662708 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:26.662610 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:26.878582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:26.878548 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:26.891750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:26.891684 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:27.441462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:27.441425 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 01:56:27.488736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:27.481876 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:28.446739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:28.445817 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:29.322612 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:29.322577 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:30.015279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:30.015250 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:30.553267 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:30.553232 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:30.878130 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:30.878044 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:31.768318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:31.768288 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:32.436522 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.436493 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 01:56:32.531836 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.531806 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf"] Oct 31 01:56:32.538617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.538586 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.547377 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.547350 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf"] Oct 31 01:56:32.547699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.547679 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2e6bfd\"/\"auth-test-model-2e6bfd-kserve-self-signed-certs\"" Oct 31 01:56:32.547901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.547884 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2e6bfd\"/\"openshift-service-ca.crt\"" Oct 31 01:56:32.548101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.548084 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-2e6bfd\"/\"kube-root-ca.crt\"" Oct 31 01:56:32.548325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.548308 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-2e6bfd\"/\"default-dockercfg-mg8q4\"" Oct 31 01:56:32.573764 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.573659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:32.682953 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.682921 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tmp-dir\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.683140 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.683118 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-model-cache\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.683195 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.683171 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tls-certs\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.683243 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.683226 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-home\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.683289 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.683278 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-dshm\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.683325 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.683304 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2p5\" (UniqueName: \"kubernetes.io/projected/0409ec2f-b150-48ee-b9ad-06e35d18c67a-kube-api-access-hr2p5\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.723023 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.713224 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:32.785740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.785614 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tmp-dir\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.786258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.785922 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-model-cache\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.786258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.785993 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tls-certs\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.786258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.786028 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-home\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.786258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.786077 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-dshm\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.786258 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.786117 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2p5\" (UniqueName: \"kubernetes.io/projected/0409ec2f-b150-48ee-b9ad-06e35d18c67a-kube-api-access-hr2p5\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.802737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.786972 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tmp-dir\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.802737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.787230 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-model-cache\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.802737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.787876 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-home\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.806731 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.804919 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-dshm\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.815421 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.815396 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2p5\" (UniqueName: \"kubernetes.io/projected/0409ec2f-b150-48ee-b9ad-06e35d18c67a-kube-api-access-hr2p5\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.835367 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.825671 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tls-certs\") pod \"auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:32.867305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.867266 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:32.877390 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:32.877103 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:33.179799 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:33.179766 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf"] Oct 31 01:56:33.180751 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:56:33.180703 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0409ec2f_b150_48ee_b9ad_06e35d18c67a.slice/crio-12be76c998a9746880c1004af50ad72f3d746aa915514709aeaea1dd5b0be8bb WatchSource:0}: Error finding container 12be76c998a9746880c1004af50ad72f3d746aa915514709aeaea1dd5b0be8bb: Status 404 returned error can't find the container with id 12be76c998a9746880c1004af50ad72f3d746aa915514709aeaea1dd5b0be8bb Oct 31 01:56:33.447709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:33.447611 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:33.487395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:33.487356 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" event={"ID":"0409ec2f-b150-48ee-b9ad-06e35d18c67a","Type":"ContainerStarted","Data":"b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7"} Oct 31 01:56:33.487617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:33.487408 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" event={"ID":"0409ec2f-b150-48ee-b9ad-06e35d18c67a","Type":"ContainerStarted","Data":"12be76c998a9746880c1004af50ad72f3d746aa915514709aeaea1dd5b0be8bb"} Oct 31 01:56:33.512183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:33.512134 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" podStartSLOduration=1.512117795 podStartE2EDuration="1.512117795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:33.506890672 +0000 UTC m=+2193.887135896" watchObservedRunningTime="2026-10-31 01:56:33.512117795 +0000 UTC m=+2193.892363018" Oct 31 01:56:33.878670 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:33.878642 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:34.201549 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:34.201472 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:34.486147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:34.486062 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:34.804134 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:56:34.804096 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:56:35.082641 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:35.082556 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:35.721642 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:35.721371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:36.332979 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:36.332946 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:37.198424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:37.197648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:38.037234 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:38.037204 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:38.211977 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:38.211944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:38.782810 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:38.782781 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:39.486050 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:39.486005 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:40.075922 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.072795 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh"] Oct 31 01:56:40.092118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.088217 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh"] Oct 31 01:56:40.092118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.088338 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.092118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.091976 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c54a1\"/\"auth-test-model-6c54a1-kserve-self-signed-certs\"" Oct 31 01:56:40.092644 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.092506 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c54a1\"/\"default-dockercfg-q8g22\"" Oct 31 01:56:40.093251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.093227 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c54a1\"/\"kube-root-ca.crt\"" Oct 31 01:56:40.093415 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.093305 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-6c54a1\"/\"openshift-service-ca.crt\"" Oct 31 01:56:40.111172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.106915 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-tmp-dir\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.111172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.106983 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119eb288-6642-40ab-ac9e-eb30122b7c7c-tls-certs\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.111172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.107140 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-model-cache\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.111172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.107404 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-home\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.111172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.107478 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2k89\" (UniqueName: \"kubernetes.io/projected/119eb288-6642-40ab-ac9e-eb30122b7c7c-kube-api-access-j2k89\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.111172 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.107877 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-dshm\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.135597 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.135568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:40.222128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.209705 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-model-cache\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.222128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.209817 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-home\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.222128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.209851 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2k89\" (UniqueName: \"kubernetes.io/projected/119eb288-6642-40ab-ac9e-eb30122b7c7c-kube-api-access-j2k89\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.222128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.209959 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-dshm\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.222128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.210084 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-tmp-dir\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.222128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.210113 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119eb288-6642-40ab-ac9e-eb30122b7c7c-tls-certs\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.225300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.225258 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-model-cache\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.225456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.225397 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-tmp-dir\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.225553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.225528 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-home\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.249743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.241837 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119eb288-6642-40ab-ac9e-eb30122b7c7c-tls-certs\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.261532 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.261501 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-dshm\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.300768 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.297448 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2k89\" (UniqueName: \"kubernetes.io/projected/119eb288-6642-40ab-ac9e-eb30122b7c7c-kube-api-access-j2k89\") pod \"auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.368799 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.364333 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:40.413575 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.413542 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:40.722047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:40.722006 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh"] Oct 31 01:56:40.722457 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:56:40.722360 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod119eb288_6642_40ab_ac9e_eb30122b7c7c.slice/crio-b58a050f8bfa79c4e410e6a997d27172222b4a6e6a2c45240a37fb85b82af97e WatchSource:0}: Error finding container b58a050f8bfa79c4e410e6a997d27172222b4a6e6a2c45240a37fb85b82af97e: Status 404 returned error can't find the container with id b58a050f8bfa79c4e410e6a997d27172222b4a6e6a2c45240a37fb85b82af97e Oct 31 01:56:41.253752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:41.253678 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:41.564442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:41.564349 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" event={"ID":"119eb288-6642-40ab-ac9e-eb30122b7c7c","Type":"ContainerStarted","Data":"f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3"} Oct 31 01:56:41.564442 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:41.564400 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" event={"ID":"119eb288-6642-40ab-ac9e-eb30122b7c7c","Type":"ContainerStarted","Data":"b58a050f8bfa79c4e410e6a997d27172222b4a6e6a2c45240a37fb85b82af97e"} Oct 31 01:56:41.607617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:41.607566 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" podStartSLOduration=1.6075516589999999 podStartE2EDuration="1.607551659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:56:41.605057295 +0000 UTC m=+2201.985302519" watchObservedRunningTime="2026-10-31 01:56:41.607551659 +0000 UTC m=+2201.987796881" Oct 31 01:56:42.063503 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:42.062538 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:42.414073 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:42.413992 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:42.756358 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:42.756324 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:42.878225 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:42.878196 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:42.897186 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:42.897158 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:43.574446 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:43.574397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:43.591072 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:43.591046 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 01:56:44.347512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:44.347477 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:45.068339 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:45.068255 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:45.838328 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:45.838289 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:46.542759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:46.542688 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:47.355129 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:47.355086 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:48.034387 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:48.034344 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:48.795173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:48.795134 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:49.095926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:49.095834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:49.650615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:49.650433 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:50.342183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:50.342148 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:50.414148 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:50.414117 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:50.427068 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:50.427044 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:50.666626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:50.666542 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 01:56:50.993930 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:50.993897 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:51.185781 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:51.185747 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:51.868611 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:51.867406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:52.615686 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:52.615645 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:53.001519 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:53.001485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:53.361547 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:53.361461 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:54.046653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:54.045388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:54.864808 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:54.864702 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:55.521571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:55.521538 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:56.419007 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:56.418970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:56.700564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:56.700488 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:57.451869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:57.451840 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:58.286302 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:58.286265 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:56:59.244434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:56:59.239974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:00.150548 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:00.150521 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:00.832801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:00.832766 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:01.542895 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:01.542858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:02.166350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:02.166313 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:02.863526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:02.863422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:03.638737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:03.638634 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:04.553253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:04.553140 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:04.790497 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:57:04.790455 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:57:05.041394 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.041362 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7"] Oct 31 01:57:05.047319 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.047297 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.050594 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.050570 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-46a67e-openshift-default-dockercfg-ttzhz\"" Oct 31 01:57:05.059118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.059094 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7"] Oct 31 01:57:05.120621 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.120574 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.120879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.120729 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.120879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.120817 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.120879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.120867 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.121128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.120889 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.121128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.120985 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.121128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.121012 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.121128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.121061 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.121343 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.121191 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpv6\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-kube-api-access-7xpv6\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.213625 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.213579 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:05.224434 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224392 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.224666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224511 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.224666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224559 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.224666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224606 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.224666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224635 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.225004 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224698 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.225004 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224745 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.225004 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224818 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.225004 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.224911 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpv6\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-kube-api-access-7xpv6\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.225754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.225683 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.233150 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.226280 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-data\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.234014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.233979 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-certs\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.234330 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.234284 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-credential-socket\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.234439 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.234348 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istiod-ca-cert\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.247238 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.247173 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-envoy\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.259682 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.254669 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpv6\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-kube-api-access-7xpv6\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.259682 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.259554 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-podinfo\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.266752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.260167 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-token\") pod \"e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.367025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.366941 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:05.654037 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.654001 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7"] Oct 31 01:57:05.655628 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:57:05.655592 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20804cd_2d74_4ca2_b037_d8ff00e003d4.slice/crio-1fa8708d249c213d0fefcc99ae832845d1524ca0af75e9dbc9384fde3ea0b553 WatchSource:0}: Error finding container 1fa8708d249c213d0fefcc99ae832845d1524ca0af75e9dbc9384fde3ea0b553: Status 404 returned error can't find the container with id 1fa8708d249c213d0fefcc99ae832845d1524ca0af75e9dbc9384fde3ea0b553 Oct 31 01:57:05.658553 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.658512 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:57:05.658653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.658599 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:57:05.658653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.658645 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 01:57:05.769283 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.769251 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" event={"ID":"e20804cd-2d74-4ca2-b037-d8ff00e003d4","Type":"ContainerStarted","Data":"ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507"} Oct 31 01:57:05.769476 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.769310 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" event={"ID":"e20804cd-2d74-4ca2-b037-d8ff00e003d4","Type":"ContainerStarted","Data":"1fa8708d249c213d0fefcc99ae832845d1524ca0af75e9dbc9384fde3ea0b553"} Oct 31 01:57:05.792363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.792308 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podStartSLOduration=0.792292345 podStartE2EDuration="792.292345ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:05.791497038 +0000 UTC m=+2226.171742286" watchObservedRunningTime="2026-10-31 01:57:05.792292345 +0000 UTC m=+2226.172537569" Oct 31 01:57:05.879822 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:05.879146 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:06.369745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:06.368881 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:06.370367 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:06.370342 5090 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:57:06.370488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:06.370389 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:57:06.647681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:06.647570 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:07.249305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:07.249271 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:07.368138 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:07.368100 5090 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:57:07.372675 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:07.372634 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:57:07.626751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:07.626635 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:07.896801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:07.850113 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:08.052743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:08.046204 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:08.367279 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:08.367246 5090 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:57:08.367783 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:08.367316 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:57:08.930743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:08.926693 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:09.368012 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:09.367981 5090 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:57:09.368406 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:09.368071 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:57:09.902605 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:09.902567 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:10.368175 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:10.368140 5090 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:57:10.368626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:10.368208 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:57:10.919443 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:10.919406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:11.368022 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:11.367982 5090 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 01:57:11.368236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:11.368050 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 01:57:11.924348 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:11.924300 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:12.504530 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:12.504495 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:12.505128 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:12.505101 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:12.506244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:12.506226 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 01:57:12.657336 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:12.657301 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:12.802084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:12.801879 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:13.386118 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:13.381486 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:13.751586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:13.751550 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:14.153804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:14.151356 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:14.867739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:14.862519 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:15.573297 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:15.573264 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:16.264187 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:16.264154 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:16.975362 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:16.975326 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:17.840291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:17.840254 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:18.355738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.345138 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5"] Oct 31 01:57:18.355738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.345400 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" podUID="fc66f2a7-2c3b-4ec9-86db-60c8276734e6" containerName="maas-api" containerID="cri-o://0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b" gracePeriod=30 Oct 31 01:57:18.432084 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.431800 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:18.588329 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.588251 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:18.728492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.728459 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:18.733097 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.733068 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:57:18.873748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.873668 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-kube-api-access-zj64d\" (UniqueName: \"kubernetes.io/projected/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-kube-api-access-zj64d\") pod \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " Oct 31 01:57:18.874278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.873871 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-openshift-service-ca\") pod \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " Oct 31 01:57:18.874278 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.873917 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-metrics-certs\") pod \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " Oct 31 01:57:18.893904 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.893847 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-metrics-certs" pod "fc66f2a7-2c3b-4ec9-86db-60c8276734e6" (UID: "fc66f2a7-2c3b-4ec9-86db-60c8276734e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:57:18.901308 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.901271 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-maas-api-tls\") pod \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\" (UID: \"fc66f2a7-2c3b-4ec9-86db-60c8276734e6\") " Oct 31 01:57:18.902305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.902277 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:18.908971 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.908921 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-openshift-service-ca" pod "fc66f2a7-2c3b-4ec9-86db-60c8276734e6" (UID: "fc66f2a7-2c3b-4ec9-86db-60c8276734e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 01:57:18.947846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.947764 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:18.948253 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.947763 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-kube-api-access-zj64d" pod "fc66f2a7-2c3b-4ec9-86db-60c8276734e6" (UID: "fc66f2a7-2c3b-4ec9-86db-60c8276734e6"). InnerVolumeSpecName "kube-api-access-zj64d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:57:18.951731 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:18.951666 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-maas-api-tls" pod "fc66f2a7-2c3b-4ec9-86db-60c8276734e6" (UID: "fc66f2a7-2c3b-4ec9-86db-60c8276734e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:57:19.005734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.003826 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:19.005734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.003864 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj64d\" (UniqueName: \"kubernetes.io/projected/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-kube-api-access-zj64d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:19.005734 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.003885 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc66f2a7-2c3b-4ec9-86db-60c8276734e6-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:19.065491 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.065459 5090 generic.go:356] "Generic (PLEG): container finished" podID="fc66f2a7-2c3b-4ec9-86db-60c8276734e6" containerID="0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b" exitCode=0 Oct 31 01:57:19.065672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.065529 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" Oct 31 01:57:19.065672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.065542 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" event={"ID":"fc66f2a7-2c3b-4ec9-86db-60c8276734e6","Type":"ContainerDied","Data":"0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b"} Oct 31 01:57:19.065672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.065619 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5" event={"ID":"fc66f2a7-2c3b-4ec9-86db-60c8276734e6","Type":"ContainerDied","Data":"90be8a15c4d7e4db4aaf8185c67e7abca7f64be8e23441fdc2d7c61c9544530a"} Oct 31 01:57:19.065672 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.065648 5090 scope.go:122] "RemoveContainer" containerID="0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b" Oct 31 01:57:19.084318 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.084274 5090 scope.go:122] "RemoveContainer" containerID="0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b" Oct 31 01:57:19.084839 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:57:19.084732 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b\": container with ID starting with 0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b not found: ID does not exist" containerID="0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b" Oct 31 01:57:19.084839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.084771 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b"} err="failed to get container status \"0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b\": rpc error: code = NotFound desc = could not find container \"0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b\": container with ID starting with 0aa82e0b088eeeff3d7ffa7bea0bf99292a5cc11b031683c7bdbd2bf0d08ce7b not found: ID does not exist" Oct 31 01:57:19.097901 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.097843 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5"] Oct 31 01:57:19.108294 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.108267 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c8b702-55bd897d7-6l5f5"] Oct 31 01:57:19.129640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.129611 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:19.514736 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.514690 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:19.748661 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:19.748629 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc66f2a7-2c3b-4ec9-86db-60c8276734e6" path="/var/lib/kubelet/pods/fc66f2a7-2c3b-4ec9-86db-60c8276734e6/volumes" Oct 31 01:57:20.173744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:20.167428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:20.857522 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:20.856316 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:21.565797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:21.565124 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:22.225755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:22.224097 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:22.921314 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:22.921278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:23.634211 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:23.634171 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:24.324495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:24.324460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:25.074818 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:25.074782 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:25.885750 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:25.883629 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:26.186595 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:26.186498 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:26.344115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:26.344084 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:26.549044 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:26.549014 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:26.705327 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:26.705293 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:27.198826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:27.198795 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:27.350060 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:27.349983 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:27.689533 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:27.689416 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:27.839741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:27.836510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:28.257534 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:28.257490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:28.425993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:28.425947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:29.258085 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:29.258055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:29.638961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:29.638888 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:29.788002 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:29.787974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:29.929352 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:29.929270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:30.075617 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:30.075585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:30.759566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:30.759530 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:31.496784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:31.496752 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:31.650630 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:31.650594 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:32.278398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:32.278366 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:32.440096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:32.440068 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:32.686627 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:32.686543 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:32.829739 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:32.826091 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:33.125201 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:33.124996 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:33.832792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:33.832756 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:34.571881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:34.571691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:34.791342 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:57:34.791306 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:57:35.263305 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:35.263274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:36.074405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:36.071454 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:36.609608 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:36.609569 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:36.779793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:36.779070 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:36.966005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:36.965911 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:37.753436 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:37.753388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:37.915541 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:37.915508 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:38.085792 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:38.079958 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:38.563316 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:38.563272 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:39.244640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:39.242873 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:39.959742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:39.952217 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:40.114936 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.114829 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm"] Oct 31 01:57:40.116822 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.115918 5090 cpu_manager.go:414] "containerMap: removing container" podUID="fc66f2a7-2c3b-4ec9-86db-60c8276734e6" containerName="maas-api" Oct 31 01:57:40.117028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.117000 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc66f2a7-2c3b-4ec9-86db-60c8276734e6" containerName="maas-api" Oct 31 01:57:40.117344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.117269 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc66f2a7-2c3b-4ec9-86db-60c8276734e6" containerName="maas-api" Oct 31 01:57:40.122777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.122709 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.127863 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.127831 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ccfa21\"/\"kube-root-ca.crt\"" Oct 31 01:57:40.128013 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.127864 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ccfa21\"/\"default-dockercfg-qtdg7\"" Oct 31 01:57:40.128013 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.127922 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ccfa21\"/\"noten-model-ccfa21-kserve-self-signed-certs\"" Oct 31 01:57:40.128181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.128157 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ccfa21\"/\"openshift-service-ca.crt\"" Oct 31 01:57:40.129136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.129117 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm"] Oct 31 01:57:40.193748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.193694 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:40.208676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.208647 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbh5\" (UniqueName: \"kubernetes.io/projected/2f950f10-272e-43e9-a033-44edfac6b697-kube-api-access-hxbh5\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.208826 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.208701 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-home\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.208887 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.208828 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-dshm\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.209000 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.208985 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-model-cache\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.209079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.209006 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f950f10-272e-43e9-a033-44edfac6b697-tls-certs\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.209079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.209022 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-tmp-dir\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.311976 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbh5\" (UniqueName: \"kubernetes.io/projected/2f950f10-272e-43e9-a033-44edfac6b697-kube-api-access-hxbh5\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.312070 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-home\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.312149 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-dshm\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.312331 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-model-cache\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.312358 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f950f10-272e-43e9-a033-44edfac6b697-tls-certs\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.312383 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-tmp-dir\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.312923 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-tmp-dir\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.315772 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.313493 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-home\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.319745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.318407 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-dshm\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.319745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.318676 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-model-cache\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.335699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.335661 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbh5\" (UniqueName: \"kubernetes.io/projected/2f950f10-272e-43e9-a033-44edfac6b697-kube-api-access-hxbh5\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.342825 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.342034 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f950f10-272e-43e9-a033-44edfac6b697-tls-certs\") pod \"noten-model-ccfa21-kserve-7f5c88497-ntrpm\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.443168 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.443132 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:40.776311 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.776272 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm"] Oct 31 01:57:40.778591 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:57:40.778553 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f950f10_272e_43e9_a033_44edfac6b697.slice/crio-b094148cff17321d5204c3128fb24f14becb906426e60c58e55295caa6e5a078 WatchSource:0}: Error finding container b094148cff17321d5204c3128fb24f14becb906426e60c58e55295caa6e5a078: Status 404 returned error can't find the container with id b094148cff17321d5204c3128fb24f14becb906426e60c58e55295caa6e5a078 Oct 31 01:57:40.853643 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:40.853603 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:41.287073 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:41.287033 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" event={"ID":"2f950f10-272e-43e9-a033-44edfac6b697","Type":"ContainerStarted","Data":"6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e"} Oct 31 01:57:41.287290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:41.287089 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" event={"ID":"2f950f10-272e-43e9-a033-44edfac6b697","Type":"ContainerStarted","Data":"b094148cff17321d5204c3128fb24f14becb906426e60c58e55295caa6e5a078"} Oct 31 01:57:41.332744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:41.327624 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" podStartSLOduration=1.3275996829999999 podStartE2EDuration="1.327599683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:57:41.319788053 +0000 UTC m=+2261.700033284" watchObservedRunningTime="2026-10-31 01:57:41.327599683 +0000 UTC m=+2261.707844910" Oct 31 01:57:41.449742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:41.446852 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:41.762800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:41.750962 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:42.582246 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:42.582206 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:43.400756 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:43.396952 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:43.896767 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:43.896709 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm"] Oct 31 01:57:43.961067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:43.961040 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:44.129694 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:44.129661 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:44.288738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:44.273640 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:44.457361 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:44.457326 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:44.596196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:44.596089 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:44.812965 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:44.812931 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:44.969981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:44.969883 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:45.570746 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:45.568759 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:46.318759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:46.317375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:46.951847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:46.951808 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:47.100428 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:47.100392 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:47.260839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:47.260809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:47.941080 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:47.941049 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:48.642277 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:48.642227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:49.340655 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:49.340616 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:49.757291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:49.757255 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:50.099835 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:50.099707 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:50.828606 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:50.828568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:51.047002 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:51.046970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:51.528350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:51.528310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:52.274289 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:52.273489 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:52.956220 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:52.956168 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:53.648173 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:53.647941 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:54.296376 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:54.296340 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:54.978618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:54.976814 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:55.658058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:55.658018 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:56.361109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:56.361050 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:57.083741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:57.082166 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:57.698669 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:57.698636 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:58.468032 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:58.467998 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:59.206850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.206806 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:57:59.367854 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.367812 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" podUID="2f950f10-272e-43e9-a033-44edfac6b697" containerName="main" containerID="cri-o://6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e" gracePeriod=45 Oct 31 01:57:59.659079 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.659053 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:57:59.724556 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.724522 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-dshm\") pod \"2f950f10-272e-43e9-a033-44edfac6b697\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " Oct 31 01:57:59.724744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.724629 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-tmp-dir\") pod \"2f950f10-272e-43e9-a033-44edfac6b697\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " Oct 31 01:57:59.724744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.724678 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-home\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-home\") pod \"2f950f10-272e-43e9-a033-44edfac6b697\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " Oct 31 01:57:59.724868 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.724811 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f950f10-272e-43e9-a033-44edfac6b697-kube-api-access-hxbh5\" (UniqueName: \"kubernetes.io/projected/2f950f10-272e-43e9-a033-44edfac6b697-kube-api-access-hxbh5\") pod \"2f950f10-272e-43e9-a033-44edfac6b697\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " Oct 31 01:57:59.724934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.724907 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f950f10-272e-43e9-a033-44edfac6b697-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f950f10-272e-43e9-a033-44edfac6b697-tls-certs\") pod \"2f950f10-272e-43e9-a033-44edfac6b697\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " Oct 31 01:57:59.724993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.724931 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-tmp-dir" pod "2f950f10-272e-43e9-a033-44edfac6b697" (UID: "2f950f10-272e-43e9-a033-44edfac6b697"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:59.725050 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.725020 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-home" pod "2f950f10-272e-43e9-a033-44edfac6b697" (UID: "2f950f10-272e-43e9-a033-44edfac6b697"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:59.725271 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.725244 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-model-cache\") pod \"2f950f10-272e-43e9-a033-44edfac6b697\" (UID: \"2f950f10-272e-43e9-a033-44edfac6b697\") " Oct 31 01:57:59.725620 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.725476 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-model-cache" pod "2f950f10-272e-43e9-a033-44edfac6b697" (UID: "2f950f10-272e-43e9-a033-44edfac6b697"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:59.726134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.726110 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:59.726245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.726139 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:59.726245 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.726157 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:59.727189 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.727165 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-dshm" pod "2f950f10-272e-43e9-a033-44edfac6b697" (UID: "2f950f10-272e-43e9-a033-44edfac6b697"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 01:57:59.727427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.727404 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f950f10-272e-43e9-a033-44edfac6b697-kube-api-access-hxbh5" pod "2f950f10-272e-43e9-a033-44edfac6b697" (UID: "2f950f10-272e-43e9-a033-44edfac6b697"). InnerVolumeSpecName "kube-api-access-hxbh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:57:59.728385 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.728362 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f950f10-272e-43e9-a033-44edfac6b697-tls-certs" pod "2f950f10-272e-43e9-a033-44edfac6b697" (UID: "2f950f10-272e-43e9-a033-44edfac6b697"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 01:57:59.827512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.827487 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxbh5\" (UniqueName: \"kubernetes.io/projected/2f950f10-272e-43e9-a033-44edfac6b697-kube-api-access-hxbh5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:59.827512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.827513 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f950f10-272e-43e9-a033-44edfac6b697-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:59.827681 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.827522 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f950f10-272e-43e9-a033-44edfac6b697-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:57:59.859406 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:57:59.854172 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:00.468270 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.468230 5090 generic.go:356] "Generic (PLEG): container finished" podID="2f950f10-272e-43e9-a033-44edfac6b697" containerID="6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e" exitCode=0 Oct 31 01:58:00.468270 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.468276 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" Oct 31 01:58:00.468842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.468281 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" event={"ID":"2f950f10-272e-43e9-a033-44edfac6b697","Type":"ContainerDied","Data":"6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e"} Oct 31 01:58:00.468842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.468329 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm" event={"ID":"2f950f10-272e-43e9-a033-44edfac6b697","Type":"ContainerDied","Data":"b094148cff17321d5204c3128fb24f14becb906426e60c58e55295caa6e5a078"} Oct 31 01:58:00.468842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.468349 5090 scope.go:122] "RemoveContainer" containerID="6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e" Oct 31 01:58:00.470137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.469789 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:00.507175 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.507144 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm"] Oct 31 01:58:00.512052 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.512029 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ccfa21/noten-model-ccfa21-kserve-7f5c88497-ntrpm"] Oct 31 01:58:00.512242 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.512222 5090 scope.go:122] "RemoveContainer" containerID="6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e" Oct 31 01:58:00.512656 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:00.512637 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e\": container with ID starting with 6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e not found: ID does not exist" containerID="6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e" Oct 31 01:58:00.512737 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:00.512667 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e"} err="failed to get container status \"6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e\": rpc error: code = NotFound desc = could not find container \"6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e\": container with ID starting with 6738de2b345e44cd04ba3b3eb9875711cb285c25f623f1145903a0bc22f6335e not found: ID does not exist" Oct 31 01:58:01.190344 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:01.190217 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:01.749300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:01.749267 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f950f10-272e-43e9-a033-44edfac6b697" path="/var/lib/kubelet/pods/2f950f10-272e-43e9-a033-44edfac6b697/volumes" Oct 31 01:58:01.842111 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:01.842077 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:01.979071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:01.979034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:02.693473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:02.693438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:02.898456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:02.898096 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:03.404703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:03.404669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:04.176365 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:04.176325 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:04.793426 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:04.793387 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:58:04.863317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:04.863272 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:05.548479 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:05.544200 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:06.255878 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:06.244160 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:06.992626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:06.992591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:07.748813 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:07.748781 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:08.443702 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:08.443648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:09.222801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:09.222758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:09.969892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:09.969857 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:10.756281 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:10.756225 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:11.559469 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:11.559436 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:12.254177 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:12.254125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:12.976422 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:12.976385 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:13.966910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:13.966873 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:14.861570 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:14.861527 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:15.531598 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:15.531563 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:16.150821 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:16.150784 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:16.769752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:16.769684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:17.445424 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:17.445394 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:18.149101 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:18.149060 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:18.843744 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:18.843681 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:19.385771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:19.385252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:19.596779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:19.596708 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:20.500981 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:20.500946 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:21.197088 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:21.195772 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:21.894293 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:21.882794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:22.666473 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:22.666331 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:23.346798 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:23.346680 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:24.074741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:24.069679 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:24.763745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:24.763676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:25.548596 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:25.548558 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:26.243264 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:26.243228 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:26.925786 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:26.925752 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:27.343187 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.343154 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:27.412067 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.412037 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11422-009566"] Oct 31 01:58:27.412862 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.412842 5090 cpu_manager.go:414] "containerMap: removing container" podUID="2f950f10-272e-43e9-a033-44edfac6b697" containerName="main" Oct 31 01:58:27.412930 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.412865 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f950f10-272e-43e9-a033-44edfac6b697" containerName="main" Oct 31 01:58:27.413089 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.413077 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f950f10-272e-43e9-a033-44edfac6b697" containerName="main" Oct 31 01:58:27.416952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.416932 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-009566" Oct 31 01:58:27.422804 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.422778 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11422-009566"] Oct 31 01:58:27.431095 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.431071 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmw2f\"" Oct 31 01:58:27.599004 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.598921 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lg7s\" (UniqueName: \"kubernetes.io/projected/c867a463-4c4a-487c-91ff-f92ed2cc730f-kube-api-access-9lg7s\") pod \"test-curl-11422-009566\" (UID: \"c867a463-4c4a-487c-91ff-f92ed2cc730f\") " pod="opendatahub/test-curl-11422-009566" Oct 31 01:58:27.700196 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.700161 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lg7s\" (UniqueName: \"kubernetes.io/projected/c867a463-4c4a-487c-91ff-f92ed2cc730f-kube-api-access-9lg7s\") pod \"test-curl-11422-009566\" (UID: \"c867a463-4c4a-487c-91ff-f92ed2cc730f\") " pod="opendatahub/test-curl-11422-009566" Oct 31 01:58:27.711413 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.711390 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lg7s\" (UniqueName: \"kubernetes.io/projected/c867a463-4c4a-487c-91ff-f92ed2cc730f-kube-api-access-9lg7s\") pod \"test-curl-11422-009566\" (UID: \"c867a463-4c4a-487c-91ff-f92ed2cc730f\") " pod="opendatahub/test-curl-11422-009566" Oct 31 01:58:27.755313 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.755281 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-009566" Oct 31 01:58:27.756968 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:27.756939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:28.081775 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:28.081748 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11422-009566"] Oct 31 01:58:28.086782 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:58:28.086748 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc867a463_4c4a_487c_91ff_f92ed2cc730f.slice/crio-3d070d21221aaee1266595ed1c990bc84dd4d748eef7e0e3ae8f3bf64d0c0c2e WatchSource:0}: Error finding container 3d070d21221aaee1266595ed1c990bc84dd4d748eef7e0e3ae8f3bf64d0c0c2e: Status 404 returned error can't find the container with id 3d070d21221aaee1266595ed1c990bc84dd4d748eef7e0e3ae8f3bf64d0c0c2e Oct 31 01:58:28.609474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:28.607928 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:28.768094 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:28.768057 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11422-009566" event={"ID":"c867a463-4c4a-487c-91ff-f92ed2cc730f","Type":"ContainerStarted","Data":"4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416"} Oct 31 01:58:28.768094 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:28.768102 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11422-009566" event={"ID":"c867a463-4c4a-487c-91ff-f92ed2cc730f","Type":"ContainerStarted","Data":"3d070d21221aaee1266595ed1c990bc84dd4d748eef7e0e3ae8f3bf64d0c0c2e"} Oct 31 01:58:28.788408 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:28.788358 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11422-009566" podStartSLOduration=1.7883411580000002 podStartE2EDuration="1.788341158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:28.784401111 +0000 UTC m=+2309.164646353" watchObservedRunningTime="2026-10-31 01:58:28.788341158 +0000 UTC m=+2309.168586382" Oct 31 01:58:29.264892 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:29.264525 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:29.527653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:29.527563 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11422-009566"] Oct 31 01:58:29.531778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:29.531755 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11422-009566"] Oct 31 01:58:29.801779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:29.801685 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:29.943130 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:29.943101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:30.109797 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:30.103591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:30.250290 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:30.250256 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:30.688108 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:30.688076 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:30.794269 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:30.794228 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11422-009566" podUID="c867a463-4c4a-487c-91ff-f92ed2cc730f" containerName="test-curl-11422-009566" containerID="cri-o://4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416" gracePeriod=2 Oct 31 01:58:30.834447 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:30.834415 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:30.981801 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:30.981689 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:31.227569 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:31.227533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:31.573163 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:31.573133 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:31.896790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:31.892894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:32.574296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:32.574195 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:33.018668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.018635 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-009566" Oct 31 01:58:33.149696 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.149609 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c867a463-4c4a-487c-91ff-f92ed2cc730f-kube-api-access-9lg7s\" (UniqueName: \"kubernetes.io/projected/c867a463-4c4a-487c-91ff-f92ed2cc730f-kube-api-access-9lg7s\") pod \"c867a463-4c4a-487c-91ff-f92ed2cc730f\" (UID: \"c867a463-4c4a-487c-91ff-f92ed2cc730f\") " Oct 31 01:58:33.151906 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.151881 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c867a463-4c4a-487c-91ff-f92ed2cc730f-kube-api-access-9lg7s" pod "c867a463-4c4a-487c-91ff-f92ed2cc730f" (UID: "c867a463-4c4a-487c-91ff-f92ed2cc730f"). InnerVolumeSpecName "kube-api-access-9lg7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:33.258740 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.253975 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lg7s\" (UniqueName: \"kubernetes.io/projected/c867a463-4c4a-487c-91ff-f92ed2cc730f-kube-api-access-9lg7s\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:58:33.273784 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.273668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:33.750738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.750687 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c867a463-4c4a-487c-91ff-f92ed2cc730f" path="/var/lib/kubelet/pods/c867a463-4c4a-487c-91ff-f92ed2cc730f/volumes" Oct 31 01:58:33.817093 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.817062 5090 generic.go:356] "Generic (PLEG): container finished" podID="c867a463-4c4a-487c-91ff-f92ed2cc730f" containerID="4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416" exitCode=137 Oct 31 01:58:33.817257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.817099 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11422-009566" Oct 31 01:58:33.817257 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.817123 5090 scope.go:122] "RemoveContainer" containerID="4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416" Oct 31 01:58:33.830047 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.830023 5090 scope.go:122] "RemoveContainer" containerID="4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416" Oct 31 01:58:33.830316 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:33.830298 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416\": container with ID starting with 4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416 not found: ID does not exist" containerID="4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416" Oct 31 01:58:33.830359 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.830326 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416"} err="failed to get container status \"4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416\": rpc error: code = NotFound desc = could not find container \"4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416\": container with ID starting with 4786f9b15feda6f3e340814e0c8d4998e50c840d762e0c20e6e685d6de248416 not found: ID does not exist" Oct 31 01:58:33.952524 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:33.951995 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:34.459137 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:34.459104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:34.603770 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:34.603738 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:34.761578 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:34.761545 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:34.792789 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:34.792753 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:58:35.556106 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:35.556068 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:35.736254 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:35.736201 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:36.025027 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:36.024984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:36.585974 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:36.585939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:37.295134 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:37.295085 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:38.184180 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:38.184130 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:38.773938 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:38.773910 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:38.929296 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:38.929261 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:39.181065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.180959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:39.652187 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.648259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:39.751175 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.751139 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11419-236a81"] Oct 31 01:58:39.752191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.752164 5090 cpu_manager.go:414] "containerMap: removing container" podUID="c867a463-4c4a-487c-91ff-f92ed2cc730f" containerName="test-curl-11422-009566" Oct 31 01:58:39.752191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.752188 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c867a463-4c4a-487c-91ff-f92ed2cc730f" containerName="test-curl-11422-009566" Oct 31 01:58:39.752431 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.752414 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="c867a463-4c4a-487c-91ff-f92ed2cc730f" containerName="test-curl-11422-009566" Oct 31 01:58:39.757514 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.757477 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-236a81"] Oct 31 01:58:39.757666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.757577 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-236a81" Oct 31 01:58:39.762456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.762435 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmw2f\"" Oct 31 01:58:39.797754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.797681 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjcsg\" (UniqueName: \"kubernetes.io/projected/654170f9-6ef2-41b0-9144-2fad7a25a3c3-kube-api-access-wjcsg\") pod \"test-curl-11419-236a81\" (UID: \"654170f9-6ef2-41b0-9144-2fad7a25a3c3\") " pod="opendatahub/test-curl-11419-236a81" Oct 31 01:58:39.902788 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.902678 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjcsg\" (UniqueName: \"kubernetes.io/projected/654170f9-6ef2-41b0-9144-2fad7a25a3c3-kube-api-access-wjcsg\") pod \"test-curl-11419-236a81\" (UID: \"654170f9-6ef2-41b0-9144-2fad7a25a3c3\") " pod="opendatahub/test-curl-11419-236a81" Oct 31 01:58:39.914993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:39.914964 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjcsg\" (UniqueName: \"kubernetes.io/projected/654170f9-6ef2-41b0-9144-2fad7a25a3c3-kube-api-access-wjcsg\") pod \"test-curl-11419-236a81\" (UID: \"654170f9-6ef2-41b0-9144-2fad7a25a3c3\") " pod="opendatahub/test-curl-11419-236a81" Oct 31 01:58:40.073660 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:40.073629 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-236a81" Oct 31 01:58:40.353513 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:40.353486 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-236a81"] Oct 31 01:58:40.354640 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:58:40.354584 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod654170f9_6ef2_41b0_9144_2fad7a25a3c3.slice/crio-cdcd1f1b442cf511fdf40d44a979a4dfbc65f84c9ec49cbcfe7b5697e6c79be3 WatchSource:0}: Error finding container cdcd1f1b442cf511fdf40d44a979a4dfbc65f84c9ec49cbcfe7b5697e6c79be3: Status 404 returned error can't find the container with id cdcd1f1b442cf511fdf40d44a979a4dfbc65f84c9ec49cbcfe7b5697e6c79be3 Oct 31 01:58:40.439287 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:40.439215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:40.909699 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:40.909664 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-236a81" event={"ID":"654170f9-6ef2-41b0-9144-2fad7a25a3c3","Type":"ContainerStarted","Data":"0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce"} Oct 31 01:58:40.910228 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:40.909749 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-236a81" event={"ID":"654170f9-6ef2-41b0-9144-2fad7a25a3c3","Type":"ContainerStarted","Data":"cdcd1f1b442cf511fdf40d44a979a4dfbc65f84c9ec49cbcfe7b5697e6c79be3"} Oct 31 01:58:40.928208 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:40.928164 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11419-236a81" podStartSLOduration=1.928151293 podStartE2EDuration="1.928151293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:40.924183544 +0000 UTC m=+2321.304428769" watchObservedRunningTime="2026-10-31 01:58:40.928151293 +0000 UTC m=+2321.308396516" Oct 31 01:58:41.081738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.080101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:41.721677 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.721488 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11419-236a81"] Oct 31 01:58:41.725512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.725483 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11419-236a81"] Oct 31 01:58:41.838104 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.834652 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:41.893755 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.893704 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11419-185384"] Oct 31 01:58:41.894583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.894560 5090 cpu_manager.go:414] "containerMap: removing container" podUID="654170f9-6ef2-41b0-9144-2fad7a25a3c3" containerName="test-curl-11419-236a81" Oct 31 01:58:41.894583 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.894580 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654170f9-6ef2-41b0-9144-2fad7a25a3c3" containerName="test-curl-11419-236a81" Oct 31 01:58:41.895145 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.895121 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="654170f9-6ef2-41b0-9144-2fad7a25a3c3" containerName="test-curl-11419-236a81" Oct 31 01:58:41.899393 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.899366 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-185384" Oct 31 01:58:41.904602 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:41.904573 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-185384"] Oct 31 01:58:42.035063 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.035021 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7t9h\" (UniqueName: \"kubernetes.io/projected/1d15c1bf-c4cd-4aec-9275-85bef63df9fd-kube-api-access-m7t9h\") pod \"test-curl-11419-185384\" (UID: \"1d15c1bf-c4cd-4aec-9275-85bef63df9fd\") " pod="opendatahub/test-curl-11419-185384" Oct 31 01:58:42.136832 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.136803 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7t9h\" (UniqueName: \"kubernetes.io/projected/1d15c1bf-c4cd-4aec-9275-85bef63df9fd-kube-api-access-m7t9h\") pod \"test-curl-11419-185384\" (UID: \"1d15c1bf-c4cd-4aec-9275-85bef63df9fd\") " pod="opendatahub/test-curl-11419-185384" Oct 31 01:58:42.157934 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.157900 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7t9h\" (UniqueName: \"kubernetes.io/projected/1d15c1bf-c4cd-4aec-9275-85bef63df9fd-kube-api-access-m7t9h\") pod \"test-curl-11419-185384\" (UID: \"1d15c1bf-c4cd-4aec-9275-85bef63df9fd\") " pod="opendatahub/test-curl-11419-185384" Oct 31 01:58:42.219096 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.219067 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-185384" Oct 31 01:58:42.501779 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.501744 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-185384"] Oct 31 01:58:42.502853 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:58:42.502827 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d15c1bf_c4cd_4aec_9275_85bef63df9fd.slice/crio-0c6e20f6e8e117a684f20c37b5b890556f5703b955277a9400eed56f77b78f55 WatchSource:0}: Error finding container 0c6e20f6e8e117a684f20c37b5b890556f5703b955277a9400eed56f77b78f55: Status 404 returned error can't find the container with id 0c6e20f6e8e117a684f20c37b5b890556f5703b955277a9400eed56f77b78f55 Oct 31 01:58:42.633315 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.633291 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:42.931309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.931213 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-185384" event={"ID":"1d15c1bf-c4cd-4aec-9275-85bef63df9fd","Type":"ContainerStarted","Data":"d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112"} Oct 31 01:58:42.931309 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.931269 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-185384" event={"ID":"1d15c1bf-c4cd-4aec-9275-85bef63df9fd","Type":"ContainerStarted","Data":"0c6e20f6e8e117a684f20c37b5b890556f5703b955277a9400eed56f77b78f55"} Oct 31 01:58:42.931555 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.931388 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11419-236a81" podUID="654170f9-6ef2-41b0-9144-2fad7a25a3c3" containerName="test-curl-11419-236a81" containerID="cri-o://0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce" gracePeriod=2 Oct 31 01:58:42.973145 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:42.973094 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11419-185384" podStartSLOduration=1.973079609 podStartE2EDuration="1.973079609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:42.971247558 +0000 UTC m=+2323.351492777" watchObservedRunningTime="2026-10-31 01:58:42.973079609 +0000 UTC m=+2323.353324831" Oct 31 01:58:43.324162 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.319918 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:43.731952 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.731869 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11419-185384"] Oct 31 01:58:43.737748 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.737701 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11419-185384"] Oct 31 01:58:43.898303 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.898269 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11419-314a63"] Oct 31 01:58:43.899038 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.899019 5090 cpu_manager.go:414] "containerMap: removing container" podUID="1d15c1bf-c4cd-4aec-9275-85bef63df9fd" containerName="test-curl-11419-185384" Oct 31 01:58:43.899116 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.899038 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d15c1bf-c4cd-4aec-9275-85bef63df9fd" containerName="test-curl-11419-185384" Oct 31 01:58:43.899180 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.899170 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d15c1bf-c4cd-4aec-9275-85bef63df9fd" containerName="test-curl-11419-185384" Oct 31 01:58:43.903175 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.903155 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:43.910855 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.910830 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-314a63"] Oct 31 01:58:43.981054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:43.981010 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzpn\" (UniqueName: \"kubernetes.io/projected/62cddc2f-d921-44da-a583-d62bf7b80baa-kube-api-access-xwzpn\") pod \"test-curl-11419-314a63\" (UID: \"62cddc2f-d921-44da-a583-d62bf7b80baa\") " pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:44.012568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.012534 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:44.082367 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.082322 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzpn\" (UniqueName: \"kubernetes.io/projected/62cddc2f-d921-44da-a583-d62bf7b80baa-kube-api-access-xwzpn\") pod \"test-curl-11419-314a63\" (UID: \"62cddc2f-d921-44da-a583-d62bf7b80baa\") " pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:44.094407 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.094381 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzpn\" (UniqueName: \"kubernetes.io/projected/62cddc2f-d921-44da-a583-d62bf7b80baa-kube-api-access-xwzpn\") pod \"test-curl-11419-314a63\" (UID: \"62cddc2f-d921-44da-a583-d62bf7b80baa\") " pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:44.220156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.220125 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:44.571218 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.571189 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-314a63"] Oct 31 01:58:44.572321 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:58:44.572291 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62cddc2f_d921_44da_a583_d62bf7b80baa.slice/crio-bd4b5daabce384de6ea3cdad555534069ed1f3bff8eb6c5b0c2bf2ca410c623a WatchSource:0}: Error finding container bd4b5daabce384de6ea3cdad555534069ed1f3bff8eb6c5b0c2bf2ca410c623a: Status 404 returned error can't find the container with id bd4b5daabce384de6ea3cdad555534069ed1f3bff8eb6c5b0c2bf2ca410c623a Oct 31 01:58:44.679274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.679237 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt"] Oct 31 01:58:44.690183 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.684854 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.698024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.697996 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt"] Oct 31 01:58:44.698236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.698219 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-5555cb-serving-cert\"" Oct 31 01:58:44.769932 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.769908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:44.801315 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.801282 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9qd\" (UniqueName: \"kubernetes.io/projected/8afb6d7a-d18e-400c-b16a-0a7296ae1313-kube-api-access-9t9qd\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.801467 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.801322 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.801529 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.801507 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.801684 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.801661 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afb6d7a-d18e-400c-b16a-0a7296ae1313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.903255 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.903166 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afb6d7a-d18e-400c-b16a-0a7296ae1313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.903435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.903359 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9qd\" (UniqueName: \"kubernetes.io/projected/8afb6d7a-d18e-400c-b16a-0a7296ae1313-kube-api-access-9t9qd\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.903435 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.903395 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.903604 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.903588 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.904094 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.904061 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afb6d7a-d18e-400c-b16a-0a7296ae1313-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.907489 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.907460 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.907586 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.907491 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.909492 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.909462 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:44.916512 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.916490 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9qd\" (UniqueName: \"kubernetes.io/projected/8afb6d7a-d18e-400c-b16a-0a7296ae1313-kube-api-access-9t9qd\") pod \"maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:44.947869 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.947834 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-314a63" event={"ID":"62cddc2f-d921-44da-a583-d62bf7b80baa","Type":"ContainerStarted","Data":"46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b"} Oct 31 01:58:44.948005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.947888 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-314a63" event={"ID":"62cddc2f-d921-44da-a583-d62bf7b80baa","Type":"ContainerStarted","Data":"bd4b5daabce384de6ea3cdad555534069ed1f3bff8eb6c5b0c2bf2ca410c623a"} Oct 31 01:58:44.948062 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.947980 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11419-185384" podUID="1d15c1bf-c4cd-4aec-9275-85bef63df9fd" containerName="test-curl-11419-185384" containerID="cri-o://d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112" gracePeriod=2 Oct 31 01:58:44.962914 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:44.962871 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11419-314a63" podStartSLOduration=1.962853377 podStartE2EDuration="1.962853377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:44.96277575 +0000 UTC m=+2325.343020990" watchObservedRunningTime="2026-10-31 01:58:44.962853377 +0000 UTC m=+2325.343098600" Oct 31 01:58:45.016837 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.016802 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:45.155984 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.155916 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-236a81" Oct 31 01:58:45.318331 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.318297 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654170f9-6ef2-41b0-9144-2fad7a25a3c3-kube-api-access-wjcsg\" (UniqueName: \"kubernetes.io/projected/654170f9-6ef2-41b0-9144-2fad7a25a3c3-kube-api-access-wjcsg\") pod \"654170f9-6ef2-41b0-9144-2fad7a25a3c3\" (UID: \"654170f9-6ef2-41b0-9144-2fad7a25a3c3\") " Oct 31 01:58:45.319416 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:58:45.319388 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afb6d7a_d18e_400c_b16a_0a7296ae1313.slice/crio-e6007fdc62ca8efb49a90746a52e3e19d1d09e0923b735f7052808863d9c0995 WatchSource:0}: Error finding container e6007fdc62ca8efb49a90746a52e3e19d1d09e0923b735f7052808863d9c0995: Status 404 returned error can't find the container with id e6007fdc62ca8efb49a90746a52e3e19d1d09e0923b735f7052808863d9c0995 Oct 31 01:58:45.322983 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.322957 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 01:58:45.323307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.323282 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt"] Oct 31 01:58:45.325743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.325700 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654170f9-6ef2-41b0-9144-2fad7a25a3c3-kube-api-access-wjcsg" pod "654170f9-6ef2-41b0-9144-2fad7a25a3c3" (UID: "654170f9-6ef2-41b0-9144-2fad7a25a3c3"). InnerVolumeSpecName "kube-api-access-wjcsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:45.425136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.425016 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjcsg\" (UniqueName: \"kubernetes.io/projected/654170f9-6ef2-41b0-9144-2fad7a25a3c3-kube-api-access-wjcsg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:58:45.748467 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.748433 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654170f9-6ef2-41b0-9144-2fad7a25a3c3" path="/var/lib/kubelet/pods/654170f9-6ef2-41b0-9144-2fad7a25a3c3/volumes" Oct 31 01:58:45.813742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.811726 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11419-314a63"] Oct 31 01:58:45.826946 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.826902 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:45.828865 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.828837 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11419-314a63"] Oct 31 01:58:45.957629 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.957604 5090 generic.go:356] "Generic (PLEG): container finished" podID="654170f9-6ef2-41b0-9144-2fad7a25a3c3" containerID="0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce" exitCode=137 Oct 31 01:58:45.957782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.957667 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-236a81" Oct 31 01:58:45.957782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.957677 5090 scope.go:122] "RemoveContainer" containerID="0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce" Oct 31 01:58:45.961071 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.961049 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" event={"ID":"8afb6d7a-d18e-400c-b16a-0a7296ae1313","Type":"ContainerStarted","Data":"e6007fdc62ca8efb49a90746a52e3e19d1d09e0923b735f7052808863d9c0995"} Oct 31 01:58:45.968025 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.968003 5090 scope.go:122] "RemoveContainer" containerID="0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce" Oct 31 01:58:45.968312 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:45.968288 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce\": container with ID starting with 0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce not found: ID does not exist" containerID="0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce" Oct 31 01:58:45.968414 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.968323 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce"} err="failed to get container status \"0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce\": rpc error: code = NotFound desc = could not find container \"0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce\": container with ID starting with 0440f7ff341b3855bd7caf97a511b6f9a207375c3b07e997e68fea874729d5ce not found: ID does not exist" Oct 31 01:58:45.991877 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.991839 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11419-dc8377"] Oct 31 01:58:45.992526 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.992511 5090 cpu_manager.go:414] "containerMap: removing container" podUID="62cddc2f-d921-44da-a583-d62bf7b80baa" containerName="test-curl-11419-314a63" Oct 31 01:58:45.992577 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.992529 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cddc2f-d921-44da-a583-d62bf7b80baa" containerName="test-curl-11419-314a63" Oct 31 01:58:45.992703 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.992691 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cddc2f-d921-44da-a583-d62bf7b80baa" containerName="test-curl-11419-314a63" Oct 31 01:58:45.997454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:45.997435 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:46.002480 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.002427 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-dc8377"] Oct 31 01:58:46.034803 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.034770 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfr2\" (UniqueName: \"kubernetes.io/projected/e7af58d1-f6dc-427f-8738-f7d24b8d4668-kube-api-access-gcfr2\") pod \"test-curl-11419-dc8377\" (UID: \"e7af58d1-f6dc-427f-8738-f7d24b8d4668\") " pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:46.136235 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.136190 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfr2\" (UniqueName: \"kubernetes.io/projected/e7af58d1-f6dc-427f-8738-f7d24b8d4668-kube-api-access-gcfr2\") pod \"test-curl-11419-dc8377\" (UID: \"e7af58d1-f6dc-427f-8738-f7d24b8d4668\") " pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:46.148005 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.147972 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfr2\" (UniqueName: \"kubernetes.io/projected/e7af58d1-f6dc-427f-8738-f7d24b8d4668-kube-api-access-gcfr2\") pod \"test-curl-11419-dc8377\" (UID: \"e7af58d1-f6dc-427f-8738-f7d24b8d4668\") " pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:46.314058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.313967 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:46.533160 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.530608 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:46.594959 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.594934 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-dc8377"] Oct 31 01:58:46.596625 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:58:46.596592 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7af58d1_f6dc_427f_8738_f7d24b8d4668.slice/crio-d8742256bdbfc965f0c16acdfea7c45026b051be0469f4bc947b9f048b3faf25 WatchSource:0}: Error finding container d8742256bdbfc965f0c16acdfea7c45026b051be0469f4bc947b9f048b3faf25: Status 404 returned error can't find the container with id d8742256bdbfc965f0c16acdfea7c45026b051be0469f4bc947b9f048b3faf25 Oct 31 01:58:46.971531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.971493 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" event={"ID":"8afb6d7a-d18e-400c-b16a-0a7296ae1313","Type":"ContainerStarted","Data":"efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e"} Oct 31 01:58:46.971937 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.971595 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:46.973291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.973266 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-dc8377" event={"ID":"e7af58d1-f6dc-427f-8738-f7d24b8d4668","Type":"ContainerStarted","Data":"a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9"} Oct 31 01:58:46.973395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.973300 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-dc8377" event={"ID":"e7af58d1-f6dc-427f-8738-f7d24b8d4668","Type":"ContainerStarted","Data":"d8742256bdbfc965f0c16acdfea7c45026b051be0469f4bc947b9f048b3faf25"} Oct 31 01:58:46.973395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.973328 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11419-314a63" podUID="62cddc2f-d921-44da-a583-d62bf7b80baa" containerName="test-curl-11419-314a63" containerID="cri-o://46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b" gracePeriod=2 Oct 31 01:58:46.992615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:46.992561 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" podStartSLOduration=2.442464531 podStartE2EDuration="2.99254396s" totalImagesPullingTime="550.079429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:46.990954356 +0000 UTC m=+2327.371199580" watchObservedRunningTime="2026-10-31 01:58:46.99254396 +0000 UTC m=+2327.372789183" Oct 31 01:58:47.007409 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.007326 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11419-dc8377" podStartSLOduration=2.007310251 podStartE2EDuration="2.007310251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:47.006556881 +0000 UTC m=+2327.386802125" watchObservedRunningTime="2026-10-31 01:58:47.007310251 +0000 UTC m=+2327.387555475" Oct 31 01:58:47.008581 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:47.008541 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11419-314a63\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="62cddc2f-d921-44da-a583-d62bf7b80baa" pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:47.169831 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.169806 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-185384" Oct 31 01:58:47.194598 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:47.194545 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11419-314a63\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="62cddc2f-d921-44da-a583-d62bf7b80baa" pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:47.204659 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.204555 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:47.266271 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.266233 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d15c1bf-c4cd-4aec-9275-85bef63df9fd-kube-api-access-m7t9h\" (UniqueName: \"kubernetes.io/projected/1d15c1bf-c4cd-4aec-9275-85bef63df9fd-kube-api-access-m7t9h\") pod \"1d15c1bf-c4cd-4aec-9275-85bef63df9fd\" (UID: \"1d15c1bf-c4cd-4aec-9275-85bef63df9fd\") " Oct 31 01:58:47.268955 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.268926 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d15c1bf-c4cd-4aec-9275-85bef63df9fd-kube-api-access-m7t9h" pod "1d15c1bf-c4cd-4aec-9275-85bef63df9fd" (UID: "1d15c1bf-c4cd-4aec-9275-85bef63df9fd"). InnerVolumeSpecName "kube-api-access-m7t9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:47.368457 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.368428 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7t9h\" (UniqueName: \"kubernetes.io/projected/1d15c1bf-c4cd-4aec-9275-85bef63df9fd-kube-api-access-m7t9h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:58:47.748427 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.748391 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d15c1bf-c4cd-4aec-9275-85bef63df9fd" path="/var/lib/kubelet/pods/1d15c1bf-c4cd-4aec-9275-85bef63df9fd/volumes" Oct 31 01:58:47.759894 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.759868 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11419-dc8377"] Oct 31 01:58:47.766149 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.766129 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11419-dc8377"] Oct 31 01:58:47.890910 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.890869 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:47.939827 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.939795 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11419-662c0f"] Oct 31 01:58:47.940633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.940611 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" containerName="test-curl-11419-dc8377" Oct 31 01:58:47.940633 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.940633 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" containerName="test-curl-11419-dc8377" Oct 31 01:58:47.940853 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.940781 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" containerName="test-curl-11419-dc8377" Oct 31 01:58:47.944842 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.944824 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:47.955103 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.955056 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-662c0f"] Oct 31 01:58:47.982006 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.981964 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgsdz\" (UniqueName: \"kubernetes.io/projected/e6ecb57d-038e-4d6c-8cc9-7680d92aee29-kube-api-access-hgsdz\") pod \"test-curl-11419-662c0f\" (UID: \"e6ecb57d-038e-4d6c-8cc9-7680d92aee29\") " pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:47.985307 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.985281 5090 generic.go:356] "Generic (PLEG): container finished" podID="1d15c1bf-c4cd-4aec-9275-85bef63df9fd" containerID="d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112" exitCode=137 Oct 31 01:58:47.985495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.985351 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-185384" Oct 31 01:58:47.985495 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.985389 5090 scope.go:122] "RemoveContainer" containerID="d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112" Oct 31 01:58:47.997766 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.997748 5090 scope.go:122] "RemoveContainer" containerID="d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112" Oct 31 01:58:47.998103 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:47.998081 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112\": container with ID starting with d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112 not found: ID does not exist" containerID="d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112" Oct 31 01:58:47.998201 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:47.998115 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112"} err="failed to get container status \"d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112\": rpc error: code = NotFound desc = could not find container \"d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112\": container with ID starting with d8947adcdc6448bbcb42eec6aeecc99a5a1cf8b39494d27ddc83a2ace38e5112 not found: ID does not exist" Oct 31 01:58:48.083564 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.083489 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgsdz\" (UniqueName: \"kubernetes.io/projected/e6ecb57d-038e-4d6c-8cc9-7680d92aee29-kube-api-access-hgsdz\") pod \"test-curl-11419-662c0f\" (UID: \"e6ecb57d-038e-4d6c-8cc9-7680d92aee29\") " pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:48.095454 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.095426 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgsdz\" (UniqueName: \"kubernetes.io/projected/e6ecb57d-038e-4d6c-8cc9-7680d92aee29-kube-api-access-hgsdz\") pod \"test-curl-11419-662c0f\" (UID: \"e6ecb57d-038e-4d6c-8cc9-7680d92aee29\") " pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:48.268274 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.268243 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:48.554317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.554286 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11419-662c0f"] Oct 31 01:58:48.555525 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:58:48.555481 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6ecb57d_038e_4d6c_8cc9_7680d92aee29.slice/crio-88b6f7e255ab2368a1ccc2944fbddb29f98fa1b295d0417c65f4052741fca0de WatchSource:0}: Error finding container 88b6f7e255ab2368a1ccc2944fbddb29f98fa1b295d0417c65f4052741fca0de: Status 404 returned error can't find the container with id 88b6f7e255ab2368a1ccc2944fbddb29f98fa1b295d0417c65f4052741fca0de Oct 31 01:58:48.677745 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.676838 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:48.997967 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.997931 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-662c0f" event={"ID":"e6ecb57d-038e-4d6c-8cc9-7680d92aee29","Type":"ContainerStarted","Data":"f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2"} Oct 31 01:58:48.998310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.997979 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11419-662c0f" event={"ID":"e6ecb57d-038e-4d6c-8cc9-7680d92aee29","Type":"ContainerStarted","Data":"88b6f7e255ab2368a1ccc2944fbddb29f98fa1b295d0417c65f4052741fca0de"} Oct 31 01:58:48.998310 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:48.998221 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11419-dc8377" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" containerName="test-curl-11419-dc8377" containerID="cri-o://a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9" gracePeriod=2 Oct 31 01:58:49.014816 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:49.014789 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11419-dc8377\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:49.015508 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.015472 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11419-662c0f" podStartSLOduration=2.015460476 podStartE2EDuration="2.015460476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:58:49.012621719 +0000 UTC m=+2329.392866959" watchObservedRunningTime="2026-10-31 01:58:49.015460476 +0000 UTC m=+2329.395705699" Oct 31 01:58:49.191024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.191001 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:49.193078 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:49.193050 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11419-dc8377\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:49.305171 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.305142 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cddc2f-d921-44da-a583-d62bf7b80baa-kube-api-access-xwzpn\" (UniqueName: \"kubernetes.io/projected/62cddc2f-d921-44da-a583-d62bf7b80baa-kube-api-access-xwzpn\") pod \"62cddc2f-d921-44da-a583-d62bf7b80baa\" (UID: \"62cddc2f-d921-44da-a583-d62bf7b80baa\") " Oct 31 01:58:49.307261 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.307236 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cddc2f-d921-44da-a583-d62bf7b80baa-kube-api-access-xwzpn" pod "62cddc2f-d921-44da-a583-d62bf7b80baa" (UID: "62cddc2f-d921-44da-a583-d62bf7b80baa"). InnerVolumeSpecName "kube-api-access-xwzpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:49.373743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.370283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:49.406653 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.406619 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzpn\" (UniqueName: \"kubernetes.io/projected/62cddc2f-d921-44da-a583-d62bf7b80baa-kube-api-access-xwzpn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:58:49.744881 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:49.744850 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11419-dc8377\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:49.749066 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.749039 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62cddc2f-d921-44da-a583-d62bf7b80baa" path="/var/lib/kubelet/pods/62cddc2f-d921-44da-a583-d62bf7b80baa/volumes" Oct 31 01:58:49.790098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.790073 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11419-662c0f"] Oct 31 01:58:49.794778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:49.794755 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11419-662c0f"] Oct 31 01:58:50.005844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:50.005762 5090 generic.go:356] "Generic (PLEG): container finished" podID="62cddc2f-d921-44da-a583-d62bf7b80baa" containerID="46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b" exitCode=137 Oct 31 01:58:50.005844 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:50.005823 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-314a63" Oct 31 01:58:50.006306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:50.005877 5090 scope.go:122] "RemoveContainer" containerID="46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b" Oct 31 01:58:50.016058 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:50.016033 5090 scope.go:122] "RemoveContainer" containerID="46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b" Oct 31 01:58:50.016339 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:50.016318 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b\": container with ID starting with 46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b not found: ID does not exist" containerID="46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b" Oct 31 01:58:50.016389 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:50.016355 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b"} err="failed to get container status \"46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b\": rpc error: code = NotFound desc = could not find container \"46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b\": container with ID starting with 46a11384e488746dd97de8758e41df809c1636eb187629cf681cd69461e2379b not found: ID does not exist" Oct 31 01:58:50.170974 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:50.166609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:50.872777 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:50.858433 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:51.016156 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.016125 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11419-662c0f" podUID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" containerName="test-curl-11419-662c0f" containerID="cri-o://f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2" gracePeriod=2 Oct 31 01:58:51.018126 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:51.018094 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11419-662c0f\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:51.103990 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.103955 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:51.188142 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.188113 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:51.189883 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:51.189856 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11419-662c0f\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:51.335507 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.335475 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7af58d1-f6dc-427f-8738-f7d24b8d4668-kube-api-access-gcfr2\" (UniqueName: \"kubernetes.io/projected/e7af58d1-f6dc-427f-8738-f7d24b8d4668-kube-api-access-gcfr2\") pod \"e7af58d1-f6dc-427f-8738-f7d24b8d4668\" (UID: \"e7af58d1-f6dc-427f-8738-f7d24b8d4668\") " Oct 31 01:58:51.337883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.337856 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7af58d1-f6dc-427f-8738-f7d24b8d4668-kube-api-access-gcfr2" pod "e7af58d1-f6dc-427f-8738-f7d24b8d4668" (UID: "e7af58d1-f6dc-427f-8738-f7d24b8d4668"). InnerVolumeSpecName "kube-api-access-gcfr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:51.437420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.437344 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcfr2\" (UniqueName: \"kubernetes.io/projected/e7af58d1-f6dc-427f-8738-f7d24b8d4668-kube-api-access-gcfr2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:58:51.588741 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.581259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:51.729654 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.729573 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:51.748589 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:51.748562 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" path="/var/lib/kubelet/pods/e7af58d1-f6dc-427f-8738-f7d24b8d4668/volumes" Oct 31 01:58:52.024024 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.023993 5090 generic.go:356] "Generic (PLEG): container finished" podID="e7af58d1-f6dc-427f-8738-f7d24b8d4668" containerID="a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9" exitCode=137 Oct 31 01:58:52.024498 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.024043 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-dc8377" Oct 31 01:58:52.024498 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.024094 5090 scope.go:122] "RemoveContainer" containerID="a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9" Oct 31 01:58:52.027418 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.027394 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:52.036947 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.036929 5090 scope.go:122] "RemoveContainer" containerID="a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9" Oct 31 01:58:52.037242 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:52.037223 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9\": container with ID starting with a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9 not found: ID does not exist" containerID="a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9" Oct 31 01:58:52.037300 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.037251 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9"} err="failed to get container status \"a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9\": rpc error: code = NotFound desc = could not find container \"a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9\": container with ID starting with a54f8d7cb4f29467a81028e307291ed46ff36714f4310f7ab3b445ef6e9afad9 not found: ID does not exist" Oct 31 01:58:52.166752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.161692 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:52.551795 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.550114 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:52.993236 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:52.993206 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 01:58:53.264284 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:53.260691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:53.312441 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:53.312412 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:53.389861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:53.389828 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ecb57d-038e-4d6c-8cc9-7680d92aee29-kube-api-access-hgsdz\" (UniqueName: \"kubernetes.io/projected/e6ecb57d-038e-4d6c-8cc9-7680d92aee29-kube-api-access-hgsdz\") pod \"e6ecb57d-038e-4d6c-8cc9-7680d92aee29\" (UID: \"e6ecb57d-038e-4d6c-8cc9-7680d92aee29\") " Oct 31 01:58:53.392117 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:53.392090 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ecb57d-038e-4d6c-8cc9-7680d92aee29-kube-api-access-hgsdz" pod "e6ecb57d-038e-4d6c-8cc9-7680d92aee29" (UID: "e6ecb57d-038e-4d6c-8cc9-7680d92aee29"). InnerVolumeSpecName "kube-api-access-hgsdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 01:58:53.492317 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:53.492286 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgsdz\" (UniqueName: \"kubernetes.io/projected/e6ecb57d-038e-4d6c-8cc9-7680d92aee29-kube-api-access-hgsdz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 01:58:53.753351 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:53.753317 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" path="/var/lib/kubelet/pods/e6ecb57d-038e-4d6c-8cc9-7680d92aee29/volumes" Oct 31 01:58:53.932751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:53.930318 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:54.051356 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:54.051270 5090 generic.go:356] "Generic (PLEG): container finished" podID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" containerID="f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2" exitCode=137 Oct 31 01:58:54.051572 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:54.051560 5090 scope.go:122] "RemoveContainer" containerID="f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2" Oct 31 01:58:54.051754 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:54.051735 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11419-662c0f" Oct 31 01:58:54.064484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:54.064460 5090 scope.go:122] "RemoveContainer" containerID="f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2" Oct 31 01:58:54.065006 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:58:54.064976 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2\": container with ID starting with f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2 not found: ID does not exist" containerID="f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2" Oct 31 01:58:54.065113 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:54.065016 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2"} err="failed to get container status \"f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2\": rpc error: code = NotFound desc = could not find container \"f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2\": container with ID starting with f3da79a8cb01b50b64e9521be56c3dfad0aa862f94ff8d7ea11bf1316c443ba2 not found: ID does not exist" Oct 31 01:58:54.729635 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:54.729600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:55.605763 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:55.605691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:55.779474 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:55.779442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:56.381742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:56.378162 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:56.719015 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:56.718929 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:56.909738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:56.909276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:57.394846 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:57.394812 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:58.160847 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:58.160769 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:58.846872 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:58.846831 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:59.558824 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:59.558781 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:58:59.719827 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:58:59.719797 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:00.195475 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:00.195440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:00.346445 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:00.346417 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:00.656569 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:00.656537 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:01.076324 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:01.076295 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:01.230392 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:01.230249 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:01.897966 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:01.897930 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:02.632861 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:02.632778 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:03.406406 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:03.406354 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:04.151241 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:04.150234 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:04.360166 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:04.360135 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:04.507167 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:04.507127 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:04.791642 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:59:04.791548 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:59:04.978771 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:04.978008 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:05.565450 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:05.565419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:05.722147 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:05.722115 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:06.357873 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.357838 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm"] Oct 31 01:59:06.358935 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.358906 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" containerName="test-curl-11419-662c0f" Oct 31 01:59:06.358935 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.358934 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" containerName="test-curl-11419-662c0f" Oct 31 01:59:06.359191 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.359170 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ecb57d-038e-4d6c-8cc9-7680d92aee29" containerName="test-curl-11419-662c0f" Oct 31 01:59:06.370072 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.370033 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.373350 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.373329 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm"] Oct 31 01:59:06.379370 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.377964 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ac7e0d-serving-cert\"" Oct 31 01:59:06.454398 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.454369 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:06.484341 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.484306 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bfd4e-2278-46a6-ba07-fb9651d5766e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.484341 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.484344 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-metrics-certs\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.484535 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.484512 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.484626 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.484611 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74m8n\" (UniqueName: \"kubernetes.io/projected/141bfd4e-2278-46a6-ba07-fb9651d5766e-kube-api-access-74m8n\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.586014 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.585975 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74m8n\" (UniqueName: \"kubernetes.io/projected/141bfd4e-2278-46a6-ba07-fb9651d5766e-kube-api-access-74m8n\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.586388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.586134 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bfd4e-2278-46a6-ba07-fb9651d5766e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.586388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.586167 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-metrics-certs\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.586388 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.586304 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.586941 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.586913 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bfd4e-2278-46a6-ba07-fb9651d5766e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.588993 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.588975 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.589115 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.589093 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-metrics-certs\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.598338 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.598309 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74m8n\" (UniqueName: \"kubernetes.io/projected/141bfd4e-2278-46a6-ba07-fb9651d5766e-kube-api-access-74m8n\") pod \"maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:06.606897 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.606867 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:06.690091 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:06.689993 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:07.010531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:07.010470 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm"] Oct 31 01:59:07.026266 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:59:07.026226 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141bfd4e_2278_46a6_ba07_fb9651d5766e.slice/crio-8c02aedc363edac44147b53dbbf39dc4730b73eae0e4dfb80864cdaf07e43c86 WatchSource:0}: Error finding container 8c02aedc363edac44147b53dbbf39dc4730b73eae0e4dfb80864cdaf07e43c86: Status 404 returned error can't find the container with id 8c02aedc363edac44147b53dbbf39dc4730b73eae0e4dfb80864cdaf07e43c86 Oct 31 01:59:07.165610 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:07.165555 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" event={"ID":"141bfd4e-2278-46a6-ba07-fb9651d5766e","Type":"ContainerStarted","Data":"8c02aedc363edac44147b53dbbf39dc4730b73eae0e4dfb80864cdaf07e43c86"} Oct 31 01:59:07.364839 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:07.364694 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:08.066244 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:08.066156 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:08.174640 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:08.174581 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" event={"ID":"141bfd4e-2278-46a6-ba07-fb9651d5766e","Type":"ContainerStarted","Data":"940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965"} Oct 31 01:59:08.175109 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:08.174818 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:08.197587 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:08.197539 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" podStartSLOduration=1.7035059430000001 podStartE2EDuration="2.197525522s" totalImagesPullingTime="494.019579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:08.196099081 +0000 UTC m=+2348.576344320" watchObservedRunningTime="2026-10-31 01:59:08.197525522 +0000 UTC m=+2348.577770745" Oct 31 01:59:08.743659 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:08.743623 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:09.381098 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:09.381010 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:10.071856 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:10.071816 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:10.457094 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:10.457010 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:10.604742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:10.604141 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:10.874913 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:10.872167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:11.864482 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:11.864442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:12.546742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:12.537034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:13.219875 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:13.219411 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:13.955916 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:13.955192 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:14.186515 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:14.186488 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 01:59:14.558193 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:14.558158 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:15.161463 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.161431 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692"] Oct 31 01:59:15.167179 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.167153 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.189985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.188709 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-89e955-serving-cert\"" Oct 31 01:59:15.189985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.189462 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.189985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.189541 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jbr\" (UniqueName: \"kubernetes.io/projected/fc746fcd-8be9-4540-96fd-7e55187869be-kube-api-access-c9jbr\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.189985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.189606 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc746fcd-8be9-4540-96fd-7e55187869be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.189985 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.189683 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.194774 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.194750 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692"] Oct 31 01:59:15.259883 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.259852 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:15.314778 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.314725 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.315126 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.315105 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jbr\" (UniqueName: \"kubernetes.io/projected/fc746fcd-8be9-4540-96fd-7e55187869be-kube-api-access-c9jbr\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.315299 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.315285 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc746fcd-8be9-4540-96fd-7e55187869be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.315502 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.315486 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.316515 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:59:15.314913 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-serving-cert: secret "maas-api-e2e-shared-b-w1-89e955-serving-cert" not found Oct 31 01:59:15.316786 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:59:15.316769 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls podName:fc746fcd-8be9-4540-96fd-7e55187869be nodeName:}" failed. No retries permitted until 2026-10-31 01:59:15.816742579 +0000 UTC m=+2356.196987799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls") pod "maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" (UID: "fc746fcd-8be9-4540-96fd-7e55187869be") : secret "maas-api-e2e-shared-b-w1-89e955-serving-cert" not found Oct 31 01:59:15.320650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.317824 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc746fcd-8be9-4540-96fd-7e55187869be-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.320650 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.320463 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.329751 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.329697 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jbr\" (UniqueName: \"kubernetes.io/projected/fc746fcd-8be9-4540-96fd-7e55187869be-kube-api-access-c9jbr\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.395419 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.395382 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:15.542618 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.542592 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:15.825803 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.824136 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:15.856888 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:15.856853 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:16.103705 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:16.103628 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:16.168747 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:16.168469 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:16.396782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:16.396758 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692"] Oct 31 01:59:16.399250 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:59:16.399225 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc746fcd_8be9_4540_96fd_7e55187869be.slice/crio-1e63fa25084faa662946bfc2675b7f58f261f76b0763520ba122967c89b6c8d4 WatchSource:0}: Error finding container 1e63fa25084faa662946bfc2675b7f58f261f76b0763520ba122967c89b6c8d4: Status 404 returned error can't find the container with id 1e63fa25084faa662946bfc2675b7f58f261f76b0763520ba122967c89b6c8d4 Oct 31 01:59:16.869414 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:16.869375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:17.253730 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:17.253677 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" event={"ID":"fc746fcd-8be9-4540-96fd-7e55187869be","Type":"ContainerStarted","Data":"7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1"} Oct 31 01:59:17.253916 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:17.253773 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:17.253916 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:17.253797 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" event={"ID":"fc746fcd-8be9-4540-96fd-7e55187869be","Type":"ContainerStarted","Data":"1e63fa25084faa662946bfc2675b7f58f261f76b0763520ba122967c89b6c8d4"} Oct 31 01:59:17.273345 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:17.273290 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" podStartSLOduration=1.802922611 podStartE2EDuration="2.273274013s" totalImagesPullingTime="470.351402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:17.272450243 +0000 UTC m=+2357.652695483" watchObservedRunningTime="2026-10-31 01:59:17.273274013 +0000 UTC m=+2357.653519236" Oct 31 01:59:17.509571 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:17.509479 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:18.195519 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:18.195483 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:18.851468 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:18.850617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:19.680761 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:19.680709 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:20.343410 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:20.343369 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:20.847490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:20.847458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:21.055999 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:21.055963 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:21.720918 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:21.720888 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:22.280040 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:22.279993 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:22.624133 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:22.624055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:22.800614 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:22.800576 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:23.001709 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:23.000262 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:23.277862 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:23.277690 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 01:59:23.543266 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:23.543142 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:24.369819 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:24.369776 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:25.064470 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:25.064426 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:26.059615 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:26.059549 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:26.427944 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:26.427839 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:26.585850 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:26.585805 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:26.970135 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:26.970095 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:27.647462 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:27.647432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:27.785793 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:27.785744 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:28.359666 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:28.359496 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:29.211484 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:29.211454 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:29.704558 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:29.704529 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:30.009306 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:30.009273 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:30.683958 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:30.683922 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:31.369028 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:31.368993 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:32.087184 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:32.087029 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:32.241865 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:32.241831 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:32.613001 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:32.612973 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:32.754360 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:32.754263 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:33.040961 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:33.040923 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:33.667363 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:33.667283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:34.346376 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:34.346341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:34.790052 ip-10-0-141-145 kubenswrapper[5090]: E1031 01:59:34.790014 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 01:59:35.229949 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:35.229875 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:36.029487 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:36.029448 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:36.755918 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:36.755886 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:36.986738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:36.986138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:37.661136 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:37.661098 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:38.026229 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:38.026196 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:38.254752 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:38.252844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:38.561782 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:38.561746 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:39.095054 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.094998 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv"] Oct 31 01:59:39.100251 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.100223 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.108743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.108693 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-0b6abf-serving-cert\"" Oct 31 01:59:39.118926 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.117989 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv"] Oct 31 01:59:39.145568 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.145535 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.145879 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.145854 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-metrics-certs\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.146124 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.146095 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec35109-1f78-4101-a506-66372eb75e36-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.146378 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.146351 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfv2v\" (UniqueName: \"kubernetes.io/projected/fec35109-1f78-4101-a506-66372eb75e36-kube-api-access-bfv2v\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.196852 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.196826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:39.248260 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.248228 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.248405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.248275 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-metrics-certs\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.248405 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.248367 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec35109-1f78-4101-a506-66372eb75e36-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.248528 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.248434 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfv2v\" (UniqueName: \"kubernetes.io/projected/fec35109-1f78-4101-a506-66372eb75e36-kube-api-access-bfv2v\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.249065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.249043 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec35109-1f78-4101-a506-66372eb75e36-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.250972 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.250951 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-maas-api-tls\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.251065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.250986 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-metrics-certs\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.295743 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.284408 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfv2v\" (UniqueName: \"kubernetes.io/projected/fec35109-1f78-4101-a506-66372eb75e36-kube-api-access-bfv2v\") pod \"maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.337566 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.337536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:39.428420 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.427662 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:39.794890 ip-10-0-141-145 kubenswrapper[5090]: W1031 01:59:39.794220 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec35109_1f78_4101_a506_66372eb75e36.slice/crio-46ac9cb7137db706318c6193e4e5a631a0dba0acfd45ea2d4b6ee4d6e3401758 WatchSource:0}: Error finding container 46ac9cb7137db706318c6193e4e5a631a0dba0acfd45ea2d4b6ee4d6e3401758: Status 404 returned error can't find the container with id 46ac9cb7137db706318c6193e4e5a631a0dba0acfd45ea2d4b6ee4d6e3401758 Oct 31 01:59:39.803582 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:39.803546 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv"] Oct 31 01:59:40.036565 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:40.036455 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:40.466158 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:40.466121 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" event={"ID":"fec35109-1f78-4101-a506-66372eb75e36","Type":"ContainerStarted","Data":"45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b"} Oct 31 01:59:40.466544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:40.466218 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" event={"ID":"fec35109-1f78-4101-a506-66372eb75e36","Type":"ContainerStarted","Data":"46ac9cb7137db706318c6193e4e5a631a0dba0acfd45ea2d4b6ee4d6e3401758"} Oct 31 01:59:40.466544 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:40.466264 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:40.488780 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:40.488695 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" podStartSLOduration=1.067501317 podStartE2EDuration="1.488682049s" totalImagesPullingTime="421.180732ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:59:40.486469657 +0000 UTC m=+2380.866714895" watchObservedRunningTime="2026-10-31 01:59:40.488682049 +0000 UTC m=+2380.868927272" Oct 31 01:59:40.820632 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:40.820589 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:41.441456 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:41.441409 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:42.130676 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:42.130645 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:42.764202 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:42.764117 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:43.620742 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:43.613823 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:44.359632 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:44.359590 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:45.047790 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:45.047188 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:45.731965 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:45.731827 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:46.502057 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:46.502022 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 01:59:46.535962 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:46.533116 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:47.152832 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:47.152798 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:47.865881 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:47.864978 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:48.534065 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:48.534023 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:49.144630 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:49.144588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:49.285668 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:49.285626 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:50.050483 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:50.050447 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:50.767759 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:50.767665 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:51.189395 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:51.189272 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:51.512228 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:51.512008 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:52.149031 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:52.148995 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:52.945291 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:52.945258 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:53.824701 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:53.818569 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:54.542800 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:54.542463 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:55.165616 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:55.165586 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:55.842490 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:55.842455 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:56.665738 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:56.663256 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:56.841181 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:56.841151 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:57.341680 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:57.341644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:58.009971 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:58.009835 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:58.151466 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:58.151432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:58.422122 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:58.422012 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:58.743857 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:58.735744 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:58.907383 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:58.907350 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:59.050531 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:59.050439 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:59.642488 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:59.642443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 01:59:59.805874 ip-10-0-141-145 kubenswrapper[5090]: I1031 01:59:59.805801 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:00.258988 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:00.258958 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:00:00.266835 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:00.266793 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:00:00.272788 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:00.272763 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:00:00.283245 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:00.283214 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:00:00.762401 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:00.762370 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:00.914348 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:00.914306 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:01.564002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:01.563968 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:02.391503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:02.391468 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:03.012178 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:03.012082 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:03.830137 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:03.830055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:04.611224 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:04.608207 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:04.793012 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:04.792971 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:00:05.481376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:05.476257 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:06.142469 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:06.142432 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:06.891468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:06.889817 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:07.261346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:07.252778 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:07.689797 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:07.687399 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:08.535657 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:08.535614 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:08.691711 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:08.691675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:09.296830 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:09.282564 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:09.583594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:09.583496 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:09.729233 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:09.729202 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:09.964568 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:09.964487 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc"] Oct 31 02:00:10.091319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:10.091259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:10.421571 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:10.421533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:11.034684 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:11.034646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:11.905335 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:11.905302 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:13.113168 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:13.113128 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:13.928911 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:13.928869 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:14.076863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:14.076824 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:14.249106 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:14.249075 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:14.730798 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:14.730492 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:15.326911 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:15.326875 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:15.611325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:15.610794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:15.760238 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:15.760184 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:16.556140 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:16.556101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:17.362052 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:17.361997 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:18.158289 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:18.158235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:18.849120 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:18.849086 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:19.597135 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:19.597100 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:19.604757 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:19.604704 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf"] Oct 31 02:00:19.767773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:19.767743 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:20.591406 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:20.591368 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:21.310816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:21.310777 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:21.468206 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:21.468172 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:21.789671 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:21.789635 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:22.246843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:22.246813 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:22.409145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:22.409111 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:23.341259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:23.341216 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:24.107899 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:24.107865 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:24.999191 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:24.999152 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" podUID="9c95bc40-df72-46ba-9b00-a725595908ec" containerName="main" containerID="cri-o://4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8" gracePeriod=45 Oct 31 02:00:25.039676 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.036559 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:25.144383 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.144341 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" podUID="9c95bc40-df72-46ba-9b00-a725595908ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.132:8000/ready\": dial tcp 10.132.0.132:8000: connect: connection refused" Oct 31 02:00:25.303667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.303642 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 02:00:25.448457 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.448424 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c95bc40-df72-46ba-9b00-a725595908ec-kube-api-access-q8d4x\" (UniqueName: \"kubernetes.io/projected/9c95bc40-df72-46ba-9b00-a725595908ec-kube-api-access-q8d4x\") pod \"9c95bc40-df72-46ba-9b00-a725595908ec\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " Oct 31 02:00:25.448647 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.448494 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c95bc40-df72-46ba-9b00-a725595908ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c95bc40-df72-46ba-9b00-a725595908ec-tls-certs\") pod \"9c95bc40-df72-46ba-9b00-a725595908ec\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " Oct 31 02:00:25.448647 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.448525 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-home\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-home\") pod \"9c95bc40-df72-46ba-9b00-a725595908ec\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " Oct 31 02:00:25.448647 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.448614 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-dshm\") pod \"9c95bc40-df72-46ba-9b00-a725595908ec\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " Oct 31 02:00:25.448873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.448771 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-model-cache\") pod \"9c95bc40-df72-46ba-9b00-a725595908ec\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " Oct 31 02:00:25.448873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.448832 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-tmp-dir\") pod \"9c95bc40-df72-46ba-9b00-a725595908ec\" (UID: \"9c95bc40-df72-46ba-9b00-a725595908ec\") " Oct 31 02:00:25.448990 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.448869 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-home" pod "9c95bc40-df72-46ba-9b00-a725595908ec" (UID: "9c95bc40-df72-46ba-9b00-a725595908ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:25.449093 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.449051 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-model-cache" pod "9c95bc40-df72-46ba-9b00-a725595908ec" (UID: "9c95bc40-df72-46ba-9b00-a725595908ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:25.449159 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.449122 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-tmp-dir" pod "9c95bc40-df72-46ba-9b00-a725595908ec" (UID: "9c95bc40-df72-46ba-9b00-a725595908ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:25.449792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.449767 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:25.449884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.449797 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:25.449884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.449812 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:25.451185 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.451161 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c95bc40-df72-46ba-9b00-a725595908ec-tls-certs" pod "9c95bc40-df72-46ba-9b00-a725595908ec" (UID: "9c95bc40-df72-46ba-9b00-a725595908ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:00:25.451185 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.451170 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-dshm" pod "9c95bc40-df72-46ba-9b00-a725595908ec" (UID: "9c95bc40-df72-46ba-9b00-a725595908ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:25.451327 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.451236 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c95bc40-df72-46ba-9b00-a725595908ec-kube-api-access-q8d4x" pod "9c95bc40-df72-46ba-9b00-a725595908ec" (UID: "9c95bc40-df72-46ba-9b00-a725595908ec"). InnerVolumeSpecName "kube-api-access-q8d4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:25.551625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.551480 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c95bc40-df72-46ba-9b00-a725595908ec-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:25.551625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.551519 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c95bc40-df72-46ba-9b00-a725595908ec-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:25.551625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.551535 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8d4x\" (UniqueName: \"kubernetes.io/projected/9c95bc40-df72-46ba-9b00-a725595908ec-kube-api-access-q8d4x\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:25.787002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.786963 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:25.932019 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:25.931947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:26.052834 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.052800 5090 generic.go:356] "Generic (PLEG): container finished" podID="9c95bc40-df72-46ba-9b00-a725595908ec" containerID="4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8" exitCode=0 Oct 31 02:00:26.053023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.052848 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" Oct 31 02:00:26.053023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.052872 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" event={"ID":"9c95bc40-df72-46ba-9b00-a725595908ec","Type":"ContainerDied","Data":"4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8"} Oct 31 02:00:26.053023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.052926 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc" event={"ID":"9c95bc40-df72-46ba-9b00-a725595908ec","Type":"ContainerDied","Data":"2223f99e180de1e4f1ae1f14bb192f023cd7da08328f6e01e812ecfcc3a219cc"} Oct 31 02:00:26.053023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.052951 5090 scope.go:122] "RemoveContainer" containerID="4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8" Oct 31 02:00:26.064920 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.064899 5090 scope.go:122] "RemoveContainer" containerID="4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8" Oct 31 02:00:26.065200 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:26.065178 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8\": container with ID starting with 4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8 not found: ID does not exist" containerID="4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8" Oct 31 02:00:26.065272 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.065208 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8"} err="failed to get container status \"4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8\": rpc error: code = NotFound desc = could not find container \"4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8\": container with ID starting with 4ed54f0fe55cfd115e52c567d009b3bb78e376cedf031e6bb1e32cd373c6b6d8 not found: ID does not exist" Oct 31 02:00:26.070693 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.070672 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc"] Oct 31 02:00:26.075175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.075152 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed4dce/auth-test-model-ed4dce-kserve-7bf68bb745-qn8fc"] Oct 31 02:00:26.148755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.139764 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:26.872414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:26.872381 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:27.016672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:27.016617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:27.159793 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:27.156284 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:27.749911 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:27.749880 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c95bc40-df72-46ba-9b00-a725595908ec" path="/var/lib/kubelet/pods/9c95bc40-df72-46ba-9b00-a725595908ec/volumes" Oct 31 02:00:27.774211 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:27.774128 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:28.575761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:28.575665 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:29.177882 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.177835 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677"] Oct 31 02:00:29.179044 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.179015 5090 cpu_manager.go:414] "containerMap: removing container" podUID="9c95bc40-df72-46ba-9b00-a725595908ec" containerName="main" Oct 31 02:00:29.179044 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.179041 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c95bc40-df72-46ba-9b00-a725595908ec" containerName="main" Oct 31 02:00:29.179224 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.179214 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c95bc40-df72-46ba-9b00-a725595908ec" containerName="main" Oct 31 02:00:29.185211 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.185179 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.193011 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.192975 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d0c6ff-serving-cert\"" Oct 31 02:00:29.193308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.193286 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677"] Oct 31 02:00:29.231448 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.231411 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.231846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.231823 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7682\" (UniqueName: \"kubernetes.io/projected/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-kube-api-access-k7682\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.232009 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.231994 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.232227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.232212 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-metrics-certs\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.312073 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.312038 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:29.334265 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.334235 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.334432 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.334380 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7682\" (UniqueName: \"kubernetes.io/projected/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-kube-api-access-k7682\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.334432 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.334416 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.334550 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.334518 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-metrics-certs\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.335199 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.335176 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.336917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.336894 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.337082 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.337066 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-metrics-certs\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.346385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.346360 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7682\" (UniqueName: \"kubernetes.io/projected/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-kube-api-access-k7682\") pod \"maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.449491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.449408 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:29.526638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.526603 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:29.890541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:29.890498 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677"] Oct 31 02:00:30.170196 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:30.170108 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:30.199048 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:30.197763 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" event={"ID":"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6","Type":"ContainerStarted","Data":"da330bee9a4ea2104ea4168bd638b065841dfc75ab91cf36b5536f04753d66ca"} Oct 31 02:00:30.798916 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:30.791222 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:31.205841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:31.205756 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" event={"ID":"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6","Type":"ContainerStarted","Data":"2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74"} Oct 31 02:00:31.206244 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:31.205977 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:31.228915 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:31.228872 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" podStartSLOduration=1.805930875 podStartE2EDuration="2.22885759s" totalImagesPullingTime="422.926715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:31.225930109 +0000 UTC m=+2431.606175350" watchObservedRunningTime="2026-10-31 02:00:31.22885759 +0000 UTC m=+2431.609102813" Oct 31 02:00:31.763492 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:31.763456 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:32.469926 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:32.469883 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:33.104557 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:33.104512 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:33.741837 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:33.741807 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:34.456612 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:34.456426 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:34.629058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:34.629020 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" podUID="0409ec2f-b150-48ee-b9ad-06e35d18c67a" containerName="main" containerID="cri-o://b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7" gracePeriod=45 Oct 31 02:00:34.793135 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:34.793102 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:00:34.921942 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:34.921915 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 02:00:35.030094 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030004 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tls-certs\") pod \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " Oct 31 02:00:35.030260 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030132 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tmp-dir\") pod \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " Oct 31 02:00:35.030260 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030168 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-model-cache\") pod \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " Oct 31 02:00:35.030260 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030232 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-dshm\") pod \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " Oct 31 02:00:35.030435 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030278 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-home\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-home\") pod \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " Oct 31 02:00:35.030435 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030333 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0409ec2f-b150-48ee-b9ad-06e35d18c67a-kube-api-access-hr2p5\" (UniqueName: \"kubernetes.io/projected/0409ec2f-b150-48ee-b9ad-06e35d18c67a-kube-api-access-hr2p5\") pod \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\" (UID: \"0409ec2f-b150-48ee-b9ad-06e35d18c67a\") " Oct 31 02:00:35.030435 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030397 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tmp-dir" pod "0409ec2f-b150-48ee-b9ad-06e35d18c67a" (UID: "0409ec2f-b150-48ee-b9ad-06e35d18c67a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:35.030664 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030472 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-model-cache" pod "0409ec2f-b150-48ee-b9ad-06e35d18c67a" (UID: "0409ec2f-b150-48ee-b9ad-06e35d18c67a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:35.030664 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030625 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-home" pod "0409ec2f-b150-48ee-b9ad-06e35d18c67a" (UID: "0409ec2f-b150-48ee-b9ad-06e35d18c67a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:35.030927 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030910 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:35.031053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030929 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:35.031053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.030939 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:35.032517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.032488 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tls-certs" pod "0409ec2f-b150-48ee-b9ad-06e35d18c67a" (UID: "0409ec2f-b150-48ee-b9ad-06e35d18c67a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:00:35.032644 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.032579 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-dshm" pod "0409ec2f-b150-48ee-b9ad-06e35d18c67a" (UID: "0409ec2f-b150-48ee-b9ad-06e35d18c67a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:35.032899 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.032872 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0409ec2f-b150-48ee-b9ad-06e35d18c67a-kube-api-access-hr2p5" pod "0409ec2f-b150-48ee-b9ad-06e35d18c67a" (UID: "0409ec2f-b150-48ee-b9ad-06e35d18c67a"). InnerVolumeSpecName "kube-api-access-hr2p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:35.069518 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.069487 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:35.141425 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.141367 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0409ec2f-b150-48ee-b9ad-06e35d18c67a-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:35.141425 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.141410 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr2p5\" (UniqueName: \"kubernetes.io/projected/0409ec2f-b150-48ee-b9ad-06e35d18c67a-kube-api-access-hr2p5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:35.141425 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.141426 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0409ec2f-b150-48ee-b9ad-06e35d18c67a-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:35.235963 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.235926 5090 generic.go:356] "Generic (PLEG): container finished" podID="0409ec2f-b150-48ee-b9ad-06e35d18c67a" containerID="b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7" exitCode=0 Oct 31 02:00:35.236151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.235977 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" Oct 31 02:00:35.236151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.236006 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" event={"ID":"0409ec2f-b150-48ee-b9ad-06e35d18c67a","Type":"ContainerDied","Data":"b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7"} Oct 31 02:00:35.236151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.236054 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf" event={"ID":"0409ec2f-b150-48ee-b9ad-06e35d18c67a","Type":"ContainerDied","Data":"12be76c998a9746880c1004af50ad72f3d746aa915514709aeaea1dd5b0be8bb"} Oct 31 02:00:35.236151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.236072 5090 scope.go:122] "RemoveContainer" containerID="b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7" Oct 31 02:00:35.255834 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.255813 5090 scope.go:122] "RemoveContainer" containerID="b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7" Oct 31 02:00:35.256203 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:35.256175 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7\": container with ID starting with b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7 not found: ID does not exist" containerID="b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7" Oct 31 02:00:35.256323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.256213 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7"} err="failed to get container status \"b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7\": rpc error: code = NotFound desc = could not find container \"b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7\": container with ID starting with b383b910fc6f768772cc9f461acd1d161eb77110e2cb1f7d801dddc1861f4cd7 not found: ID does not exist" Oct 31 02:00:35.270414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.270385 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf"] Oct 31 02:00:35.275279 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.275258 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-2e6bfd/auth-test-model-2e6bfd-kserve-6fd66584b5-mpjjf"] Oct 31 02:00:35.749798 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.746289 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:35.773288 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:35.773253 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0409ec2f-b150-48ee-b9ad-06e35d18c67a" path="/var/lib/kubelet/pods/0409ec2f-b150-48ee-b9ad-06e35d18c67a/volumes" Oct 31 02:00:36.526085 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:36.526034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:37.176830 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:37.176734 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:37.220125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:37.220099 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:00:37.653778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:37.653741 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl"] Oct 31 02:00:37.711159 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:37.711119 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:38.009619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:38.009578 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:38.686814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:38.686387 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:39.433763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:39.433731 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:40.237778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.237744 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66"] Oct 31 02:00:40.239014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.238985 5090 cpu_manager.go:414] "containerMap: removing container" podUID="0409ec2f-b150-48ee-b9ad-06e35d18c67a" containerName="main" Oct 31 02:00:40.239014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.239011 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0409ec2f-b150-48ee-b9ad-06e35d18c67a" containerName="main" Oct 31 02:00:40.240211 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.239272 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="0409ec2f-b150-48ee-b9ad-06e35d18c67a" containerName="main" Oct 31 02:00:40.247758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.247736 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.258356 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.256549 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-bb5c51-serving-cert\"" Oct 31 02:00:40.267978 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.267952 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66"] Oct 31 02:00:40.329196 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.329138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:40.363134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.360931 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56101a31-a9b1-4b1d-9214-20d1ac327da9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.363134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.361097 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hw5m\" (UniqueName: \"kubernetes.io/projected/56101a31-a9b1-4b1d-9214-20d1ac327da9-kube-api-access-2hw5m\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.363134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.361154 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.363134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.361212 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-metrics-certs\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.462882 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.462842 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56101a31-a9b1-4b1d-9214-20d1ac327da9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.463108 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.462995 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hw5m\" (UniqueName: \"kubernetes.io/projected/56101a31-a9b1-4b1d-9214-20d1ac327da9-kube-api-access-2hw5m\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.463108 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.463039 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.463108 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.463072 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-metrics-certs\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.463287 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:40.463228 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-serving-cert: secret "maas-api-e2e-extmod-a-bb5c51-serving-cert" not found Oct 31 02:00:40.463355 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:40.463306 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls podName:56101a31-a9b1-4b1d-9214-20d1ac327da9 nodeName:}" failed. No retries permitted until 2026-10-31 02:00:40.963283575 +0000 UTC m=+2441.343528796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls") pod "maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" (UID: "56101a31-a9b1-4b1d-9214-20d1ac327da9") : secret "maas-api-e2e-extmod-a-bb5c51-serving-cert" not found Oct 31 02:00:40.463966 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.463935 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56101a31-a9b1-4b1d-9214-20d1ac327da9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.466201 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.466176 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-metrics-certs\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.475176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.475154 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hw5m\" (UniqueName: \"kubernetes.io/projected/56101a31-a9b1-4b1d-9214-20d1ac327da9-kube-api-access-2hw5m\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:40.496249 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.496213 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:40.647214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.647182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:40.975796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:40.974903 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:41.006760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:41.005892 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:41.176515 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:41.176483 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:41.254265 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:41.253331 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:41.465167 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:41.465135 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66"] Oct 31 02:00:41.467058 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:00:41.467023 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56101a31_a9b1_4b1d_9214_20d1ac327da9.slice/crio-16a32f4141e9259e49afb3b57dcd72ab88c3f25c0def9391b17e7a4a831ba9ad WatchSource:0}: Error finding container 16a32f4141e9259e49afb3b57dcd72ab88c3f25c0def9391b17e7a4a831ba9ad: Status 404 returned error can't find the container with id 16a32f4141e9259e49afb3b57dcd72ab88c3f25c0def9391b17e7a4a831ba9ad Oct 31 02:00:41.944644 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:41.944614 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:42.316563 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:42.316530 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" event={"ID":"56101a31-a9b1-4b1d-9214-20d1ac327da9","Type":"ContainerStarted","Data":"f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8"} Oct 31 02:00:42.317145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:42.316583 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" event={"ID":"56101a31-a9b1-4b1d-9214-20d1ac327da9","Type":"ContainerStarted","Data":"16a32f4141e9259e49afb3b57dcd72ab88c3f25c0def9391b17e7a4a831ba9ad"} Oct 31 02:00:42.317145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:42.316696 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:42.357972 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:42.357911 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" podStartSLOduration=1.837624279 podStartE2EDuration="2.357894498s" totalImagesPullingTime="520.270219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:00:42.335213482 +0000 UTC m=+2442.715458705" watchObservedRunningTime="2026-10-31 02:00:42.357894498 +0000 UTC m=+2442.738139747" Oct 31 02:00:42.741961 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:42.741917 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:43.478278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:43.478241 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:43.481358 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:43.481329 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh"] Oct 31 02:00:43.673738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:43.671252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:44.244451 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:44.244410 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:44.975445 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:44.975408 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:45.164074 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:45.164004 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:45.643532 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:45.643496 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:45.861973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:45.861925 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:46.660997 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:46.660941 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:47.428725 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:47.428688 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:48.330380 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:48.330348 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:00:48.398364 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:48.387414 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:49.120127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:49.120092 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:49.263545 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:49.263511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:49.638899 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:49.638869 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:49.981248 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:49.981156 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:50.330109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:50.330079 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:50.500539 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:50.500465 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:50.889632 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:50.889601 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:51.102342 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:51.102290 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:51.268686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:51.268631 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:51.431660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:51.431621 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:51.580503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:51.580429 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:51.730124 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:51.730087 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:51.902844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:51.892639 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:52.344917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:52.344870 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:52.504802 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:52.504692 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:52.694438 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:52.691504 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" podUID="01f10455-5aa7-4d95-8893-7ae79b8bfe00" containerName="main" containerID="cri-o://a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6" gracePeriod=45 Oct 31 02:00:52.760746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:52.757441 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:53.063667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.063561 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:53.240269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.240199 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 02:00:53.325420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.325386 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f10455-5aa7-4d95-8893-7ae79b8bfe00-kube-api-access-nsj9r\" (UniqueName: \"kubernetes.io/projected/01f10455-5aa7-4d95-8893-7ae79b8bfe00-kube-api-access-nsj9r\") pod \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " Oct 31 02:00:53.325614 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.325547 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-model-cache\") pod \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " Oct 31 02:00:53.325614 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.325602 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-home\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-home\") pod \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " Oct 31 02:00:53.325751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.325686 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tmp-dir\") pod \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " Oct 31 02:00:53.325814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.325756 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tls-certs\" (UniqueName: \"kubernetes.io/secret/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tls-certs\") pod \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " Oct 31 02:00:53.325814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.325788 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-dshm\") pod \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\" (UID: \"01f10455-5aa7-4d95-8893-7ae79b8bfe00\") " Oct 31 02:00:53.325922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.325905 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-home" pod "01f10455-5aa7-4d95-8893-7ae79b8bfe00" (UID: "01f10455-5aa7-4d95-8893-7ae79b8bfe00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:53.326125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.326095 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tmp-dir" pod "01f10455-5aa7-4d95-8893-7ae79b8bfe00" (UID: "01f10455-5aa7-4d95-8893-7ae79b8bfe00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:53.326419 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.326399 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-model-cache" pod "01f10455-5aa7-4d95-8893-7ae79b8bfe00" (UID: "01f10455-5aa7-4d95-8893-7ae79b8bfe00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:53.326653 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.326631 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:53.326750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.326661 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:53.326750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.326677 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:53.329078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.328685 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-dshm" pod "01f10455-5aa7-4d95-8893-7ae79b8bfe00" (UID: "01f10455-5aa7-4d95-8893-7ae79b8bfe00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:53.329078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.328996 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f10455-5aa7-4d95-8893-7ae79b8bfe00-kube-api-access-nsj9r" pod "01f10455-5aa7-4d95-8893-7ae79b8bfe00" (UID: "01f10455-5aa7-4d95-8893-7ae79b8bfe00"). InnerVolumeSpecName "kube-api-access-nsj9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:53.330300 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.330270 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tls-certs" pod "01f10455-5aa7-4d95-8893-7ae79b8bfe00" (UID: "01f10455-5aa7-4d95-8893-7ae79b8bfe00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:00:53.432034 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.431998 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f10455-5aa7-4d95-8893-7ae79b8bfe00-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:53.432034 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.432035 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01f10455-5aa7-4d95-8893-7ae79b8bfe00-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:53.432034 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.432051 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsj9r\" (UniqueName: \"kubernetes.io/projected/01f10455-5aa7-4d95-8893-7ae79b8bfe00-kube-api-access-nsj9r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:53.512101 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.512062 5090 generic.go:356] "Generic (PLEG): container finished" podID="01f10455-5aa7-4d95-8893-7ae79b8bfe00" containerID="a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6" exitCode=0 Oct 31 02:00:53.512382 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.512186 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" event={"ID":"01f10455-5aa7-4d95-8893-7ae79b8bfe00","Type":"ContainerDied","Data":"a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6"} Oct 31 02:00:53.512382 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.512217 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" event={"ID":"01f10455-5aa7-4d95-8893-7ae79b8bfe00","Type":"ContainerDied","Data":"c0ca1305012c140fa1477d22cc7f4d59b449acedece7563189c80f2903a2b0f4"} Oct 31 02:00:53.512382 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.512234 5090 scope.go:122] "RemoveContainer" containerID="a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6" Oct 31 02:00:53.512557 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.512398 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl" Oct 31 02:00:53.526679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.526639 5090 scope.go:122] "RemoveContainer" containerID="a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6" Oct 31 02:00:53.527092 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:53.527066 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6\": container with ID starting with a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6 not found: ID does not exist" containerID="a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6" Oct 31 02:00:53.527213 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.527102 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6"} err="failed to get container status \"a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6\": rpc error: code = NotFound desc = could not find container \"a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6\": container with ID starting with a2e0ff61a62211dffcecfee9af7e971647f228f3991e78ed954ba8eb0aa689f6 not found: ID does not exist" Oct 31 02:00:53.542323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.542295 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl"] Oct 31 02:00:53.547870 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.547843 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-46a67e/auth-test-model-46a67e-kserve-64896d6dbf-t7lhl"] Oct 31 02:00:53.749159 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.749127 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f10455-5aa7-4d95-8893-7ae79b8bfe00" path="/var/lib/kubelet/pods/01f10455-5aa7-4d95-8893-7ae79b8bfe00/volumes" Oct 31 02:00:53.923117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:53.923090 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:54.639323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:54.639292 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:55.498936 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:55.498909 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:56.264376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:56.264341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:57.055203 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:57.055168 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:58.339163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:58.339125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:58.530104 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:58.530056 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" podUID="119eb288-6642-40ab-ac9e-eb30122b7c7c" containerName="main" containerID="cri-o://f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3" gracePeriod=45 Oct 31 02:00:58.655739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:58.653961 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" podUID="119eb288-6642-40ab-ac9e-eb30122b7c7c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": dial tcp 10.132.0.139:8000: connect: connection refused" Oct 31 02:00:58.858336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:58.858295 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:58.936822 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:58.936797 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 02:00:59.047446 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.047403 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-dshm\") pod \"119eb288-6642-40ab-ac9e-eb30122b7c7c\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " Oct 31 02:00:59.047644 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.047538 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-model-cache\") pod \"119eb288-6642-40ab-ac9e-eb30122b7c7c\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " Oct 31 02:00:59.047644 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.047602 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-home\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-home\") pod \"119eb288-6642-40ab-ac9e-eb30122b7c7c\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " Oct 31 02:00:59.047791 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.047672 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119eb288-6642-40ab-ac9e-eb30122b7c7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/119eb288-6642-40ab-ac9e-eb30122b7c7c-tls-certs\") pod \"119eb288-6642-40ab-ac9e-eb30122b7c7c\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " Oct 31 02:00:59.047876 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.047852 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119eb288-6642-40ab-ac9e-eb30122b7c7c-kube-api-access-j2k89\" (UniqueName: \"kubernetes.io/projected/119eb288-6642-40ab-ac9e-eb30122b7c7c-kube-api-access-j2k89\") pod \"119eb288-6642-40ab-ac9e-eb30122b7c7c\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " Oct 31 02:00:59.047957 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.047922 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-tmp-dir\") pod \"119eb288-6642-40ab-ac9e-eb30122b7c7c\" (UID: \"119eb288-6642-40ab-ac9e-eb30122b7c7c\") " Oct 31 02:00:59.048019 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.047854 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-model-cache" pod "119eb288-6642-40ab-ac9e-eb30122b7c7c" (UID: "119eb288-6642-40ab-ac9e-eb30122b7c7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:59.048315 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.048273 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-tmp-dir" pod "119eb288-6642-40ab-ac9e-eb30122b7c7c" (UID: "119eb288-6642-40ab-ac9e-eb30122b7c7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:59.048442 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.048307 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-home" pod "119eb288-6642-40ab-ac9e-eb30122b7c7c" (UID: "119eb288-6642-40ab-ac9e-eb30122b7c7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:59.048981 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.048954 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.049079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.048985 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.049079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.049002 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.050871 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.050847 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119eb288-6642-40ab-ac9e-eb30122b7c7c-tls-certs" pod "119eb288-6642-40ab-ac9e-eb30122b7c7c" (UID: "119eb288-6642-40ab-ac9e-eb30122b7c7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:00:59.050957 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.050903 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119eb288-6642-40ab-ac9e-eb30122b7c7c-kube-api-access-j2k89" pod "119eb288-6642-40ab-ac9e-eb30122b7c7c" (UID: "119eb288-6642-40ab-ac9e-eb30122b7c7c"). InnerVolumeSpecName "kube-api-access-j2k89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:00:59.051056 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.051041 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-dshm" pod "119eb288-6642-40ab-ac9e-eb30122b7c7c" (UID: "119eb288-6642-40ab-ac9e-eb30122b7c7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:00:59.150599 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.150559 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119eb288-6642-40ab-ac9e-eb30122b7c7c-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.150599 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.150600 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2k89\" (UniqueName: \"kubernetes.io/projected/119eb288-6642-40ab-ac9e-eb30122b7c7c-kube-api-access-j2k89\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.150844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.150616 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119eb288-6642-40ab-ac9e-eb30122b7c7c-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:00:59.552673 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.549690 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:00:59.623486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.623447 5090 generic.go:356] "Generic (PLEG): container finished" podID="119eb288-6642-40ab-ac9e-eb30122b7c7c" containerID="f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3" exitCode=0 Oct 31 02:00:59.623694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.623496 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" Oct 31 02:00:59.623694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.623497 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" event={"ID":"119eb288-6642-40ab-ac9e-eb30122b7c7c","Type":"ContainerDied","Data":"f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3"} Oct 31 02:00:59.623694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.623538 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh" event={"ID":"119eb288-6642-40ab-ac9e-eb30122b7c7c","Type":"ContainerDied","Data":"b58a050f8bfa79c4e410e6a997d27172222b4a6e6a2c45240a37fb85b82af97e"} Oct 31 02:00:59.623694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.623560 5090 scope.go:122] "RemoveContainer" containerID="f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3" Oct 31 02:00:59.635547 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.635521 5090 scope.go:122] "RemoveContainer" containerID="f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3" Oct 31 02:00:59.635853 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:00:59.635832 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3\": container with ID starting with f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3 not found: ID does not exist" containerID="f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3" Oct 31 02:00:59.635925 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.635866 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3"} err="failed to get container status \"f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3\": rpc error: code = NotFound desc = could not find container \"f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3\": container with ID starting with f1d2e64ddd701d79fe9cf3082bfbb6d4d3b38bf64c3527c1fe8c3711a6366df3 not found: ID does not exist" Oct 31 02:00:59.647833 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.647753 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh"] Oct 31 02:00:59.652934 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.652911 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-6c54a1/auth-test-model-6c54a1-kserve-5d78b476c7-qbvrh"] Oct 31 02:00:59.717030 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.717003 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11428-a047dd"] Oct 31 02:00:59.717799 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.717784 5090 cpu_manager.go:414] "containerMap: removing container" podUID="01f10455-5aa7-4d95-8893-7ae79b8bfe00" containerName="main" Oct 31 02:00:59.717841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.717801 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f10455-5aa7-4d95-8893-7ae79b8bfe00" containerName="main" Oct 31 02:00:59.717841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.717811 5090 cpu_manager.go:414] "containerMap: removing container" podUID="119eb288-6642-40ab-ac9e-eb30122b7c7c" containerName="main" Oct 31 02:00:59.717841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.717816 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119eb288-6642-40ab-ac9e-eb30122b7c7c" containerName="main" Oct 31 02:00:59.717954 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.717936 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f10455-5aa7-4d95-8893-7ae79b8bfe00" containerName="main" Oct 31 02:00:59.717954 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.717946 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="119eb288-6642-40ab-ac9e-eb30122b7c7c" containerName="main" Oct 31 02:00:59.723151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.723127 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:00:59.726115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.726091 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11428-a047dd"] Oct 31 02:00:59.726440 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.726417 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmw2f\"" Oct 31 02:00:59.753015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.752979 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119eb288-6642-40ab-ac9e-eb30122b7c7c" path="/var/lib/kubelet/pods/119eb288-6642-40ab-ac9e-eb30122b7c7c/volumes" Oct 31 02:00:59.866036 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.865943 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l769r\" (UniqueName: \"kubernetes.io/projected/ba9ac99c-496e-45db-923a-d5e4af654b0a-kube-api-access-l769r\") pod \"test-curl-11428-a047dd\" (UID: \"ba9ac99c-496e-45db-923a-d5e4af654b0a\") " pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:00:59.967673 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.967629 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l769r\" (UniqueName: \"kubernetes.io/projected/ba9ac99c-496e-45db-923a-d5e4af654b0a-kube-api-access-l769r\") pod \"test-curl-11428-a047dd\" (UID: \"ba9ac99c-496e-45db-923a-d5e4af654b0a\") " pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:00:59.980166 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:00:59.980144 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l769r\" (UniqueName: \"kubernetes.io/projected/ba9ac99c-496e-45db-923a-d5e4af654b0a-kube-api-access-l769r\") pod \"test-curl-11428-a047dd\" (UID: \"ba9ac99c-496e-45db-923a-d5e4af654b0a\") " pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:01:00.044341 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.044311 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qmw2f\"" Oct 31 02:01:00.052556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.052533 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:01:00.179630 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.179416 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:00.334635 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.334579 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11428-a047dd"] Oct 31 02:01:00.335872 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:01:00.335839 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba9ac99c_496e_45db_923a_d5e4af654b0a.slice/crio-c52459d11cedee8885bea3081642d6c86f80ff804ad13b93a9932c1507188c93 WatchSource:0}: Error finding container c52459d11cedee8885bea3081642d6c86f80ff804ad13b93a9932c1507188c93: Status 404 returned error can't find the container with id c52459d11cedee8885bea3081642d6c86f80ff804ad13b93a9932c1507188c93 Oct 31 02:01:00.634026 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.633979 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11428-a047dd" event={"ID":"ba9ac99c-496e-45db-923a-d5e4af654b0a","Type":"ContainerStarted","Data":"d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5"} Oct 31 02:01:00.634410 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.634033 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11428-a047dd" event={"ID":"ba9ac99c-496e-45db-923a-d5e4af654b0a","Type":"ContainerStarted","Data":"c52459d11cedee8885bea3081642d6c86f80ff804ad13b93a9932c1507188c93"} Oct 31 02:01:00.649635 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.649545 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11428-a047dd" podStartSLOduration=1.649530982 podStartE2EDuration="1.649530982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:00.647998755 +0000 UTC m=+2461.028243977" watchObservedRunningTime="2026-10-31 02:01:00.649530982 +0000 UTC m=+2461.029776205" Oct 31 02:01:00.981311 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:00.980266 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:01.182844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:01.177607 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:01.409069 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:01.408977 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11428-a047dd"] Oct 31 02:01:01.416951 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:01.416918 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11428-a047dd"] Oct 31 02:01:01.751085 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:01.751051 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:02.424411 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:02.424357 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:02.624138 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:02.624104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:02.648430 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:02.648392 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11428-a047dd" podUID="ba9ac99c-496e-45db-923a-d5e4af654b0a" containerName="test-curl-11428-a047dd" containerID="cri-o://d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5" gracePeriod=2 Oct 31 02:01:03.083741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:03.082474 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:03.239743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:03.235308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:03.379796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:03.379143 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:04.148690 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:04.148654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:04.802851 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:01:04.802819 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:01:04.863997 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:04.863952 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:04.877706 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:04.877661 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:01:04.878211 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:04.878189 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9ac99c-496e-45db-923a-d5e4af654b0a-kube-api-access-l769r\" (UniqueName: \"kubernetes.io/projected/ba9ac99c-496e-45db-923a-d5e4af654b0a-kube-api-access-l769r\") pod \"ba9ac99c-496e-45db-923a-d5e4af654b0a\" (UID: \"ba9ac99c-496e-45db-923a-d5e4af654b0a\") " Oct 31 02:01:04.880694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:04.880663 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9ac99c-496e-45db-923a-d5e4af654b0a-kube-api-access-l769r" pod "ba9ac99c-496e-45db-923a-d5e4af654b0a" (UID: "ba9ac99c-496e-45db-923a-d5e4af654b0a"). InnerVolumeSpecName "kube-api-access-l769r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:01:04.980102 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:04.980031 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l769r\" (UniqueName: \"kubernetes.io/projected/ba9ac99c-496e-45db-923a-d5e4af654b0a-kube-api-access-l769r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:05.443134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.443083 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw"] Oct 31 02:01:05.444250 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.444221 5090 cpu_manager.go:414] "containerMap: removing container" podUID="ba9ac99c-496e-45db-923a-d5e4af654b0a" containerName="test-curl-11428-a047dd" Oct 31 02:01:05.444250 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.444250 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9ac99c-496e-45db-923a-d5e4af654b0a" containerName="test-curl-11428-a047dd" Oct 31 02:01:05.444508 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.444492 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9ac99c-496e-45db-923a-d5e4af654b0a" containerName="test-curl-11428-a047dd" Oct 31 02:01:05.449105 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.449070 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.457737 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.456850 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2186c3-serving-cert\"" Oct 31 02:01:05.463413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.463050 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw"] Oct 31 02:01:05.486577 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.486528 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.486823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.486591 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6967a11e-c22c-4e94-9782-81c9cd8f3cec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.486823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.486706 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxk8\" (UniqueName: \"kubernetes.io/projected/6967a11e-c22c-4e94-9782-81c9cd8f3cec-kube-api-access-llxk8\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.486982 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.486969 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-metrics-certs\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.533387 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.533361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:05.588491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.588453 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxk8\" (UniqueName: \"kubernetes.io/projected/6967a11e-c22c-4e94-9782-81c9cd8f3cec-kube-api-access-llxk8\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.588777 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.588745 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-metrics-certs\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.588892 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.588795 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.588892 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.588833 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6967a11e-c22c-4e94-9782-81c9cd8f3cec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.589620 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.589597 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6967a11e-c22c-4e94-9782-81c9cd8f3cec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.592262 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.592236 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-metrics-certs\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.592464 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.592434 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.600125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.600095 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxk8\" (UniqueName: \"kubernetes.io/projected/6967a11e-c22c-4e94-9782-81c9cd8f3cec-kube-api-access-llxk8\") pod \"maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:05.686757 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.679243 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:05.690443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.687921 5090 generic.go:356] "Generic (PLEG): container finished" podID="ba9ac99c-496e-45db-923a-d5e4af654b0a" containerID="d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5" exitCode=137 Oct 31 02:01:05.690443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.688210 5090 scope.go:122] "RemoveContainer" containerID="d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5" Oct 31 02:01:05.690443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.688364 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:01:05.715165 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:01:05.715100 5090 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11428-a047dd\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="ba9ac99c-496e-45db-923a-d5e4af654b0a" pod="opendatahub/test-curl-11428-a047dd" Oct 31 02:01:05.722124 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.721638 5090 scope.go:122] "RemoveContainer" containerID="d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5" Oct 31 02:01:05.722124 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:01:05.722051 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5\": container with ID starting with d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5 not found: ID does not exist" containerID="d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5" Oct 31 02:01:05.722124 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.722083 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5"} err="failed to get container status \"d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5\": rpc error: code = NotFound desc = could not find container \"d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5\": container with ID starting with d4a51bef7d3076b07ac39255dc3c00cf44e123c7bf67684b60d75e1f5e38cdb5 not found: ID does not exist" Oct 31 02:01:05.751307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.751277 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9ac99c-496e-45db-923a-d5e4af654b0a" path="/var/lib/kubelet/pods/ba9ac99c-496e-45db-923a-d5e4af654b0a/volumes" Oct 31 02:01:05.771491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:05.771462 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:06.062972 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:06.062941 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw"] Oct 31 02:01:06.064319 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:01:06.064289 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6967a11e_c22c_4e94_9782_81c9cd8f3cec.slice/crio-34c70a56bf518dd77e21f30030b54b6fe81f328abbffddbcd0faf58cbd8c61a6 WatchSource:0}: Error finding container 34c70a56bf518dd77e21f30030b54b6fe81f328abbffddbcd0faf58cbd8c61a6: Status 404 returned error can't find the container with id 34c70a56bf518dd77e21f30030b54b6fe81f328abbffddbcd0faf58cbd8c61a6 Oct 31 02:01:06.390574 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:06.390489 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:06.634934 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:06.634878 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:06.701113 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:06.701085 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" event={"ID":"6967a11e-c22c-4e94-9782-81c9cd8f3cec","Type":"ContainerStarted","Data":"34c70a56bf518dd77e21f30030b54b6fe81f328abbffddbcd0faf58cbd8c61a6"} Oct 31 02:01:06.795455 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:06.795419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:07.484574 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:07.484540 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:07.724297 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:07.724259 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" event={"ID":"6967a11e-c22c-4e94-9782-81c9cd8f3cec","Type":"ContainerStarted","Data":"8802078ea040e4d8c592cd6c35eee6633eb5ac21aec930ef929e1c52f1b8cb0c"} Oct 31 02:01:07.724700 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:07.724468 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:07.747446 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:07.747403 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" podStartSLOduration=2.136646586 podStartE2EDuration="2.747390458s" totalImagesPullingTime="610.743872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:07.746466937 +0000 UTC m=+2468.126712176" watchObservedRunningTime="2026-10-31 02:01:07.747390458 +0000 UTC m=+2468.127635739" Oct 31 02:01:07.921492 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:07.921461 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:08.084307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:08.081649 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:08.552970 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:08.552933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:09.182709 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.182674 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:09.273654 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.273624 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8"] Oct 31 02:01:09.278431 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.278399 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.295370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.295342 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8"] Oct 31 02:01:09.322808 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.322769 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.322997 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.322859 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.322997 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.322907 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.323128 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.323014 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63a9e50-e475-408e-887a-0d49721bd7e4-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.323128 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.323111 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.323233 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.323198 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.323290 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.323242 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zq6\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-kube-api-access-69zq6\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.323346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.323301 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.323346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.323332 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.373831 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.373794 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d0c6ff-openshift-default-dockercfg-j64m7\"" Oct 31 02:01:09.425033 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.424999 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425201 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425041 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zq6\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-kube-api-access-69zq6\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425201 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425084 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425201 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425106 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425201 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425186 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425235 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425273 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425370 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63a9e50-e475-408e-887a-0d49721bd7e4-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425545 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425434 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425745 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425910 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425792 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-data\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425910 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425897 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-credential-socket\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.425994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.425925 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-certs\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.426402 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.426380 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63a9e50-e475-408e-887a-0d49721bd7e4-istiod-ca-cert\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.427779 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.427757 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-envoy\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.427903 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.427887 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-podinfo\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.437136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.437074 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-token\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.437251 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.437192 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zq6\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-kube-api-access-69zq6\") pod \"e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.600778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.600699 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:09.906161 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.906128 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8"] Oct 31 02:01:09.906789 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:01:09.906742 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63a9e50_e475_408e_887a_0d49721bd7e4.slice/crio-a9299bb68cfe1edd1e6192c7ed93e46fc739ae3bf4f208e76da9f0a78198daca WatchSource:0}: Error finding container a9299bb68cfe1edd1e6192c7ed93e46fc739ae3bf4f208e76da9f0a78198daca: Status 404 returned error can't find the container with id a9299bb68cfe1edd1e6192c7ed93e46fc739ae3bf4f208e76da9f0a78198daca Oct 31 02:01:09.911650 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.911605 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:09.911797 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.911702 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:09.911797 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:09.911786 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:10.062814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:10.058092 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:10.748861 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:10.748822 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" event={"ID":"d63a9e50-e475-408e-887a-0d49721bd7e4","Type":"ContainerStarted","Data":"1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6"} Oct 31 02:01:10.749379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:10.749344 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" event={"ID":"d63a9e50-e475-408e-887a-0d49721bd7e4","Type":"ContainerStarted","Data":"a9299bb68cfe1edd1e6192c7ed93e46fc739ae3bf4f208e76da9f0a78198daca"} Oct 31 02:01:10.791970 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:10.791903 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podStartSLOduration=1.791884152 podStartE2EDuration="1.791884152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:10.788562668 +0000 UTC m=+2471.168807919" watchObservedRunningTime="2026-10-31 02:01:10.791884152 +0000 UTC m=+2471.172129377" Oct 31 02:01:10.924730 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:10.924650 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:11.601278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:11.601241 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:11.603606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:11.603576 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 02:01:11.603788 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:11.603629 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 02:01:11.666680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:11.666638 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:12.520068 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:12.520010 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:12.601265 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:12.601232 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 02:01:12.601528 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:12.601498 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 02:01:13.220770 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:13.220652 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:13.601184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:13.601142 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 02:01:13.601653 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:13.601223 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 02:01:13.737275 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:13.737247 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:01:14.017844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:14.007108 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:14.602015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:14.601982 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 02:01:14.602447 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:14.602051 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 02:01:14.715401 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:14.715365 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:15.461738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:15.458379 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:15.601856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:15.601796 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 02:01:15.602079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:15.601894 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 02:01:16.200742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:16.195802 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:16.608625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:16.608570 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:16.609426 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:16.609405 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:16.610229 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:16.610207 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:01:16.949649 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:16.949519 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:17.680330 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:17.680296 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:18.445740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:18.441993 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:19.147243 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:19.147198 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:19.764949 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:19.764911 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:20.469586 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:20.469548 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:21.184740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.184250 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:21.399386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.399335 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd"] Oct 31 02:01:21.404479 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.404454 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.409875 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.409854 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae5a1c-openshift-default-dockercfg-ctkcr\"" Oct 31 02:01:21.418603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.418582 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd"] Oct 31 02:01:21.490355 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490267 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.490729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490406 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wlxb\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-kube-api-access-6wlxb\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.490729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490543 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.490729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490594 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.490729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490674 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.490729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490695 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a369468-1514-41de-9a9e-4fbc627b0095-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.490909 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490847 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.490959 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490940 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.491015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.490999 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a369468-1514-41de-9a9e-4fbc627b0095-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.591798 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.591769 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.591952 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.591825 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.591952 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.591856 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a369468-1514-41de-9a9e-4fbc627b0095-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.591952 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.591899 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.591952 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.591939 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wlxb\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-kube-api-access-6wlxb\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.591992 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592017 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592094 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592118 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a369468-1514-41de-9a9e-4fbc627b0095-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592176 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-data\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592239 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-certs\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592451 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592608 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592543 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-credential-socket\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.592898 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.592874 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a369468-1514-41de-9a9e-4fbc627b0095-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.594465 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.594440 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-envoy\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.594682 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.594665 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a369468-1514-41de-9a9e-4fbc627b0095-istio-podinfo\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.603478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.603453 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-istio-token\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.603695 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.603671 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wlxb\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-kube-api-access-6wlxb\") pod \"e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.721605 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.721574 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:21.850225 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:21.850169 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:22.020325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.020297 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd"] Oct 31 02:01:22.021670 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:01:22.021640 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a369468_1514_41de_9a9e_4fbc627b0095.slice/crio-b6d2c51be4412ca2fdba94e2cd50a0d8c1f7c98cc8a7b6d4463f717809afe5a9 WatchSource:0}: Error finding container b6d2c51be4412ca2fdba94e2cd50a0d8c1f7c98cc8a7b6d4463f717809afe5a9: Status 404 returned error can't find the container with id b6d2c51be4412ca2fdba94e2cd50a0d8c1f7c98cc8a7b6d4463f717809afe5a9 Oct 31 02:01:22.024746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.024691 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:22.024837 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.024781 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:22.024837 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.024812 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:22.451967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.451876 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q"] Oct 31 02:01:22.459356 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.459182 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.486231 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.486197 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-91b6b1-serving-cert\"" Oct 31 02:01:22.500586 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.500555 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q"] Oct 31 02:01:22.504680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.504649 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-metrics-certs\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.504851 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.504784 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.504851 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.504825 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae11a79a-08da-477e-8fc4-a8593d626487-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.504982 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.504911 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wzk\" (UniqueName: \"kubernetes.io/projected/ae11a79a-08da-477e-8fc4-a8593d626487-kube-api-access-z5wzk\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.542219 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.542190 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:22.633452 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.633301 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.633879 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.633851 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae11a79a-08da-477e-8fc4-a8593d626487-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.660419 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.656974 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wzk\" (UniqueName: \"kubernetes.io/projected/ae11a79a-08da-477e-8fc4-a8593d626487-kube-api-access-z5wzk\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.660419 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.657179 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-metrics-certs\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.660419 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:01:22.657882 5090 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-serving-cert: secret "maas-api-e2e-extmod-b-91b6b1-serving-cert" not found Oct 31 02:01:22.660419 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:01:22.657985 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls podName:ae11a79a-08da-477e-8fc4-a8593d626487 nodeName:}" failed. No retries permitted until 2026-10-31 02:01:23.157966173 +0000 UTC m=+2483.538211387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls") pod "maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" (UID: "ae11a79a-08da-477e-8fc4-a8593d626487") : secret "maas-api-e2e-extmod-b-91b6b1-serving-cert" not found Oct 31 02:01:22.663842 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.663813 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae11a79a-08da-477e-8fc4-a8593d626487-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.666306 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.666277 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-metrics-certs\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.680664 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.680637 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wzk\" (UniqueName: \"kubernetes.io/projected/ae11a79a-08da-477e-8fc4-a8593d626487-kube-api-access-z5wzk\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:22.696018 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.695775 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:22.838183 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.838147 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:22.881743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.881165 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" event={"ID":"5a369468-1514-41de-9a9e-4fbc627b0095","Type":"ContainerStarted","Data":"4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555"} Oct 31 02:01:22.881743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.881326 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" event={"ID":"5a369468-1514-41de-9a9e-4fbc627b0095","Type":"ContainerStarted","Data":"b6d2c51be4412ca2fdba94e2cd50a0d8c1f7c98cc8a7b6d4463f717809afe5a9"} Oct 31 02:01:22.906950 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:22.906893 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podStartSLOduration=1.906876901 podStartE2EDuration="1.906876901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:22.90522483 +0000 UTC m=+2483.285470053" watchObservedRunningTime="2026-10-31 02:01:22.906876901 +0000 UTC m=+2483.287122124" Oct 31 02:01:23.172947 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.172672 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:23.231742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.212969 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:23.407776 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.407700 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:23.725083 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.724942 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:23.730755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.728836 5090 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 02:01:23.730755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.728901 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 02:01:23.738238 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.737130 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:23.757789 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:01:23.756689 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae11a79a_08da_477e_8fc4_a8593d626487.slice/crio-847ed9bc0eb141b93a81b0618a7904a43ec491b98efc35a016e2411bc38561b3 WatchSource:0}: Error finding container 847ed9bc0eb141b93a81b0618a7904a43ec491b98efc35a016e2411bc38561b3: Status 404 returned error can't find the container with id 847ed9bc0eb141b93a81b0618a7904a43ec491b98efc35a016e2411bc38561b3 Oct 31 02:01:23.762252 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.761174 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q"] Oct 31 02:01:23.923135 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.923099 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" event={"ID":"ae11a79a-08da-477e-8fc4-a8593d626487","Type":"ContainerStarted","Data":"847ed9bc0eb141b93a81b0618a7904a43ec491b98efc35a016e2411bc38561b3"} Oct 31 02:01:23.939937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:23.939902 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:24.193289 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:24.193249 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:24.723156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:24.723122 5090 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 02:01:24.723441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:24.723412 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 02:01:24.950655 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:24.950614 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" event={"ID":"ae11a79a-08da-477e-8fc4-a8593d626487","Type":"ContainerStarted","Data":"ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417"} Oct 31 02:01:24.951094 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:24.950682 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:24.964954 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:24.964928 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:24.969158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:24.969103 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" podStartSLOduration=2.455065872 podStartE2EDuration="2.969084627s" totalImagesPullingTime="514.018755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:24.968485573 +0000 UTC m=+2485.348730816" watchObservedRunningTime="2026-10-31 02:01:24.969084627 +0000 UTC m=+2485.349329853" Oct 31 02:01:25.517538 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:25.517497 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm"] Oct 31 02:01:25.517825 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:25.517794 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" containerID="cri-o://2cfea859e5dd5785d83edb02b83bd0bdc229deadedc45577d1ec288dd7c25fb2" gracePeriod=30 Oct 31 02:01:25.722437 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:25.722399 5090 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 02:01:25.722602 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:25.722456 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 02:01:25.813126 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:25.813030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:26.601741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:26.600702 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:26.722133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:26.722099 5090 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 02:01:26.722296 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:26.722163 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 02:01:27.391746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:27.388150 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:27.722446 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:27.722356 5090 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 02:01:27.722446 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:27.722423 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 02:01:28.138935 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:28.138901 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:28.539059 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:28.539018 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:28.723763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:28.723114 5090 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 02:01:28.723763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:28.723181 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 02:01:29.012760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:29.012728 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:29.728752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:29.728683 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:29.729156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:29.729136 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:29.729827 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:29.729806 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:01:29.768187 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:29.767297 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:29.912203 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:29.912160 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:30.117169 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:30.117134 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:30.461166 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:30.461084 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:30.712147 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:30.712034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:30.855762 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:30.854614 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:30.990369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:30.990290 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:01:31.005279 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.005233 5090 generic.go:356] "Generic (PLEG): container finished" podID="47e2d7e2-60f2-4704-9785-a65450587573" containerID="2cfea859e5dd5785d83edb02b83bd0bdc229deadedc45577d1ec288dd7c25fb2" exitCode=0 Oct 31 02:01:31.005538 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.005516 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" event={"ID":"47e2d7e2-60f2-4704-9785-a65450587573","Type":"ContainerDied","Data":"2cfea859e5dd5785d83edb02b83bd0bdc229deadedc45577d1ec288dd7c25fb2"} Oct 31 02:01:31.094800 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.094425 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165285 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e2d7e2-60f2-4704-9785-a65450587573-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e2d7e2-60f2-4704-9785-a65450587573-istiod-ca-cert\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165395 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-socket\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165460 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-envoy\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165492 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-kube-api-access-wd9g9\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-kube-api-access-wd9g9\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165596 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-istio-token\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-istio-token\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165643 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47e2d7e2-60f2-4704-9785-a65450587573-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e2d7e2-60f2-4704-9785-a65450587573-istio-podinfo\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165703 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-certs\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165776 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-credential-socket\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.165818 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-data\") pod \"47e2d7e2-60f2-4704-9785-a65450587573\" (UID: \"47e2d7e2-60f2-4704-9785-a65450587573\") " Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.167165 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-data" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.167305 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-socket" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.167406 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e2d7e2-60f2-4704-9785-a65450587573-istiod-ca-cert" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.167683 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-certs" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:01:31.169768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.169063 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-credential-socket" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:01:31.170631 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.169919 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-istio-token" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:01:31.170631 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.170478 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-kube-api-access-wd9g9" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "kube-api-access-wd9g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:01:31.171593 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.171367 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-envoy" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:01:31.171733 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.171606 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e2d7e2-60f2-4704-9785-a65450587573-istio-podinfo" pod "47e2d7e2-60f2-4704-9785-a65450587573" (UID: "47e2d7e2-60f2-4704-9785-a65450587573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:01:31.268697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268666 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e2d7e2-60f2-4704-9785-a65450587573-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268693 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268702 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268728 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268740 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e2d7e2-60f2-4704-9785-a65450587573-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268749 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268757 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e2d7e2-60f2-4704-9785-a65450587573-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268765 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd9g9\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-kube-api-access-wd9g9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.268968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.268778 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e2d7e2-60f2-4704-9785-a65450587573-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:01:31.269153 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.269093 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:31.524794 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.522572 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:31.737355 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.737325 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:31.889802 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:31.888907 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:32.067735 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:32.067584 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" event={"ID":"47e2d7e2-60f2-4704-9785-a65450587573","Type":"ContainerDied","Data":"3ac2fea97e1e75893065e796414afaa61fcffb92b32ddd066ffeac113f99bdcc"} Oct 31 02:01:32.067735 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:32.067666 5090 scope.go:122] "RemoveContainer" containerID="2cfea859e5dd5785d83edb02b83bd0bdc229deadedc45577d1ec288dd7c25fb2" Oct 31 02:01:32.068016 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:32.067901 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm" Oct 31 02:01:32.155688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:32.155657 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm"] Oct 31 02:01:32.159945 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:32.159894 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-kcvjm"] Oct 31 02:01:32.466336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:32.466235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:33.256606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:33.256530 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:33.754113 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:33.754075 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e2d7e2-60f2-4704-9785-a65450587573" path="/var/lib/kubelet/pods/47e2d7e2-60f2-4704-9785-a65450587573/volumes" Oct 31 02:01:34.043036 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:34.041386 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:34.791218 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:01:34.791186 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:01:34.886210 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:34.886174 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:35.130752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:35.130644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:35.411032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:35.410945 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:35.860473 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:35.860421 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:36.077645 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:36.077612 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:36.244129 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:36.244072 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:36.406802 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:36.406770 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:36.589792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:36.586798 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:37.166891 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:37.166855 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:37.543564 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:37.543528 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:37.747740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:37.741054 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:38.244040 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:38.244008 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:39.190125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:39.190056 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:40.047911 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:40.047873 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:40.912031 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:40.911986 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:41.570414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:41.570378 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:42.268807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:42.267908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:42.975376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:42.975337 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:43.776814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:43.776777 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:44.553670 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:44.553634 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:45.267343 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:45.267310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:45.928914 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:45.928790 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:46.580175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:46.580139 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:47.248391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:47.248355 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:47.503674 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:47.503594 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:48.048460 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.048411 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:48.138395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.138356 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr"] Oct 31 02:01:48.139128 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.139111 5090 cpu_manager.go:414] "containerMap: removing container" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" Oct 31 02:01:48.139184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.139131 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" Oct 31 02:01:48.139271 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.139262 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e2d7e2-60f2-4704-9785-a65450587573" containerName="istio-proxy" Oct 31 02:01:48.144873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.144853 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.155800 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.155775 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-91b6b1-openshift-default-dockercfg-9wq4c\"" Oct 31 02:01:48.158704 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.158681 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr"] Oct 31 02:01:48.294698 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.294668 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdrn4\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-kube-api-access-jdrn4\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.294806 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482267d2-2956-4473-af8a-112c174a7b35-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.294849 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.294965 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.295014 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.295063 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.295166 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.295206 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482267d2-2956-4473-af8a-112c174a7b35-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.295458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.295233 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397179 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397107 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397179 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397161 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482267d2-2956-4473-af8a-112c174a7b35-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397188 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397243 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdrn4\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-kube-api-access-jdrn4\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397306 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482267d2-2956-4473-af8a-112c174a7b35-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397345 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397419 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397483 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397591 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397637 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-data\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397798 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-certs\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397917 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.397958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.397931 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-credential-socket\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.398227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.398205 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482267d2-2956-4473-af8a-112c174a7b35-istiod-ca-cert\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.400394 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.400377 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-envoy\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.400759 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.400737 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482267d2-2956-4473-af8a-112c174a7b35-istio-podinfo\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.414854 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.414832 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-istio-token\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.423435 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.423413 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdrn4\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-kube-api-access-jdrn4\") pod \"e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.462729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.462682 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:48.693888 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.693811 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:48.763624 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.763601 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr"] Oct 31 02:01:48.764947 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:01:48.764920 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482267d2_2956_4473_af8a_112c174a7b35.slice/crio-5dd3cb7ac4cf468a39cf18b566376e852020f081da002b6c8e2031a9532bb6d0 WatchSource:0}: Error finding container 5dd3cb7ac4cf468a39cf18b566376e852020f081da002b6c8e2031a9532bb6d0: Status 404 returned error can't find the container with id 5dd3cb7ac4cf468a39cf18b566376e852020f081da002b6c8e2031a9532bb6d0 Oct 31 02:01:48.767971 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.767936 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:48.768068 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.768003 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:48.768068 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.768032 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:01:48.837496 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:48.837458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:49.052334 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.052293 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:49.195791 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.195736 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:49.226450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.226414 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" event={"ID":"482267d2-2956-4473-af8a-112c174a7b35","Type":"ContainerStarted","Data":"4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4"} Oct 31 02:01:49.226657 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.226474 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" event={"ID":"482267d2-2956-4473-af8a-112c174a7b35","Type":"ContainerStarted","Data":"5dd3cb7ac4cf468a39cf18b566376e852020f081da002b6c8e2031a9532bb6d0"} Oct 31 02:01:49.251348 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.251295 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podStartSLOduration=1.251279288 podStartE2EDuration="1.251279288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:01:49.249052348 +0000 UTC m=+2509.629297572" watchObservedRunningTime="2026-10-31 02:01:49.251279288 +0000 UTC m=+2509.631524547" Oct 31 02:01:49.346452 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.346372 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:49.463562 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.463529 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:49.465109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.465084 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 02:01:49.465231 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.465134 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 02:01:49.505481 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.505452 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:49.652050 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.651971 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:49.812399 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:49.812328 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:50.142919 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:50.142883 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:50.296755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:50.292668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:50.463638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:50.463144 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 02:01:50.463638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:50.463206 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 02:01:50.963853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:50.963826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:51.157295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:51.157265 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:51.536739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:51.522298 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:51.544750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:51.543054 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 02:01:51.544750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:51.543117 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 02:01:52.132874 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:52.132828 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:52.463543 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:52.463462 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 02:01:52.463543 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:52.463528 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 02:01:52.963106 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:52.963056 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:53.466366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:53.466329 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 02:01:53.466580 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:53.466389 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 02:01:53.768310 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:53.768274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:53.923332 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:53.923297 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:54.463212 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:54.463169 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 02:01:54.463850 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:54.463814 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 02:01:54.893792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:54.893751 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:55.636638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:55.636595 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:55.637303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:55.637102 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:55.639159 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:55.639138 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:01:55.708672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:55.708641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:56.638873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:56.638826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:56.802584 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:56.802552 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:57.496621 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:57.490556 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:58.280331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:58.280294 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:59.027414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:59.027385 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:01:59.187041 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:59.187008 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q"] Oct 31 02:01:59.187275 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:59.187245 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" containerID="cri-o://28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d" gracePeriod=30 Oct 31 02:01:59.730323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:01:59.730276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:00.495314 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:00.495274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:01.369947 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:01.369871 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:02.218366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:02.218329 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:02.865120 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:02.865081 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:03.629543 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:03.629507 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:04.272441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.272405 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:04.481175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.481145 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 02:02:04.503991 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.503958 5090 generic.go:356] "Generic (PLEG): container finished" podID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerID="28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d" exitCode=0 Oct 31 02:02:04.504205 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.504013 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" Oct 31 02:02:04.504205 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.504032 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" event={"ID":"006e6dab-5f1e-4c66-a2b6-42d2e5152628","Type":"ContainerDied","Data":"28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d"} Oct 31 02:02:04.504205 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.504102 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q" event={"ID":"006e6dab-5f1e-4c66-a2b6-42d2e5152628","Type":"ContainerDied","Data":"add69bd8002bc484f4dd762c30ab9e3db5fee6b360dff073998b7c01e1f1b79e"} Oct 31 02:02:04.504205 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.504134 5090 scope.go:122] "RemoveContainer" containerID="28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d" Oct 31 02:02:04.519765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.519726 5090 scope.go:122] "RemoveContainer" containerID="28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d" Oct 31 02:02:04.520113 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:02:04.520091 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d\": container with ID starting with 28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d not found: ID does not exist" containerID="28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d" Oct 31 02:02:04.520214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.520125 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d"} err="failed to get container status \"28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d\": rpc error: code = NotFound desc = could not find container \"28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d\": container with ID starting with 28a5c0a2322c84d8c0d30acf275ccc4d571bad631c96df2491967b02c2aeb09d not found: ID does not exist" Oct 31 02:02:04.611816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.611784 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-credential-socket\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.611996 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.611872 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-envoy\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.611996 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.611929 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-certs\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.612101 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612005 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-token\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-token\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.612101 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612055 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-podinfo\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.612191 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612108 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-credential-socket" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:02:04.612191 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612168 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-socket\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.612285 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612219 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istiod-ca-cert\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.612285 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612249 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-data\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.612285 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612252 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-certs" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:02:04.612460 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612396 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-kube-api-access-ncck2\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-kube-api-access-ncck2\") pod \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\" (UID: \"006e6dab-5f1e-4c66-a2b6-42d2e5152628\") " Oct 31 02:02:04.612524 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612479 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-data" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:02:04.612524 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612502 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-socket" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:02:04.612629 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.612594 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istiod-ca-cert" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:02:04.613261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.613084 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.613261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.613111 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.613261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.613125 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.613261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.613137 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.613261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.613146 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.614572 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.614542 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-envoy" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:02:04.614862 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.614846 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-kube-api-access-ncck2" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "kube-api-access-ncck2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:02:04.615014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.614993 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-token" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:02:04.615107 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.615013 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-podinfo" pod "006e6dab-5f1e-4c66-a2b6-42d2e5152628" (UID: "006e6dab-5f1e-4c66-a2b6-42d2e5152628"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:02:04.714155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.714083 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.714155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.714111 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.714155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.714121 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/006e6dab-5f1e-4c66-a2b6-42d2e5152628-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.714155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.714130 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncck2\" (UniqueName: \"kubernetes.io/projected/006e6dab-5f1e-4c66-a2b6-42d2e5152628-kube-api-access-ncck2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:02:04.791947 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:02:04.791910 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:02:04.829136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.829103 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q"] Oct 31 02:02:04.833408 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.833380 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-bkb6q"] Oct 31 02:02:04.964878 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:04.964798 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:05.749829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:05.749792 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" path="/var/lib/kubelet/pods/006e6dab-5f1e-4c66-a2b6-42d2e5152628/volumes" Oct 31 02:02:05.867037 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:05.867004 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:06.671226 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.671196 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:06.936747 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.936639 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r"] Oct 31 02:02:06.937434 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.937419 5090 cpu_manager.go:414] "containerMap: removing container" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" Oct 31 02:02:06.937434 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.937435 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" Oct 31 02:02:06.937544 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.937534 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="006e6dab-5f1e-4c66-a2b6-42d2e5152628" containerName="istio-proxy" Oct 31 02:02:06.943221 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.943201 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:06.952546 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.952508 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bb5c51-openshift-default-dockercfg-s2hdn\"" Oct 31 02:02:06.957661 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:06.957614 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r"] Oct 31 02:02:07.057057 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057021 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057057 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057062 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqdh\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-kube-api-access-nzqdh\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057275 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057192 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057275 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057245 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057362 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057297 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130cabf-4761-43c0-bfce-7b34b0aceba2-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057412 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057396 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057470 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057456 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057574 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057560 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.057624 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.057603 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159056 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159022 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130cabf-4761-43c0-bfce-7b34b0aceba2-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159225 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159082 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159225 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159134 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159225 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159176 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159225 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159206 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159451 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159310 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159451 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159337 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqdh\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-kube-api-access-nzqdh\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159547 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159668 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159640 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159747 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159675 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159786 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159743 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159788 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159868 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159837 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.159903 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.159874 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130cabf-4761-43c0-bfce-7b34b0aceba2-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.162032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.161999 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.162131 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.162114 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.173009 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.172987 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqdh\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-kube-api-access-nzqdh\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.173114 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.173075 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.259844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.259814 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:07.367988 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.367946 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:07.555367 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.555334 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r"] Oct 31 02:02:07.555914 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:02:07.555881 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb130cabf_4761_43c0_bfce_7b34b0aceba2.slice/crio-41b9887c95e95d8ad30044849487af0ac2d0148e8acf0bdf9dc825e3bba48db1 WatchSource:0}: Error finding container 41b9887c95e95d8ad30044849487af0ac2d0148e8acf0bdf9dc825e3bba48db1: Status 404 returned error can't find the container with id 41b9887c95e95d8ad30044849487af0ac2d0148e8acf0bdf9dc825e3bba48db1 Oct 31 02:02:07.559089 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.559055 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:02:07.559160 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.559125 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:02:07.559160 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:07.559155 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:02:08.090821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:08.090791 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:08.541769 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:08.541426 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" event={"ID":"b130cabf-4761-43c0-bfce-7b34b0aceba2","Type":"ContainerStarted","Data":"ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e"} Oct 31 02:02:08.541769 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:08.541495 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" event={"ID":"b130cabf-4761-43c0-bfce-7b34b0aceba2","Type":"ContainerStarted","Data":"41b9887c95e95d8ad30044849487af0ac2d0148e8acf0bdf9dc825e3bba48db1"} Oct 31 02:02:08.570911 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:08.570855 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podStartSLOduration=2.570838986 podStartE2EDuration="2.570838986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:02:08.567170396 +0000 UTC m=+2528.947415642" watchObservedRunningTime="2026-10-31 02:02:08.570838986 +0000 UTC m=+2528.951084220" Oct 31 02:02:08.861503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:08.858506 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:09.260009 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:09.259982 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:09.261831 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:09.261808 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 02:02:09.261920 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:09.261856 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 02:02:09.693804 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:09.692147 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:10.260267 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:10.260226 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 02:02:10.260652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:10.260286 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 02:02:10.363306 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:10.363243 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:11.141739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:11.141104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:11.260884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:11.260848 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 02:02:11.261366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:11.260919 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 02:02:11.358923 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:11.358890 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:11.859180 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:11.859142 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:12.260917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:12.260888 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 02:02:12.261329 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:12.260940 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 02:02:12.854482 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:12.854426 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:13.260337 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:13.260298 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 02:02:13.260533 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:13.260386 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 02:02:13.614019 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:13.613889 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:14.266515 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:14.266486 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:14.267009 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:14.266992 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:14.267841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:14.267819 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:02:14.555314 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:14.555232 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:15.453846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:15.453801 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:15.803613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:15.803582 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:16.180141 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:16.180044 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:17.105354 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:17.105307 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:17.331533 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:17.331503 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:17.917830 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:17.917791 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:18.107865 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:18.107827 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:18.431072 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:18.431035 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:19.110031 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:19.109982 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:20.146687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:20.146587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:20.991064 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:20.990945 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:21.780346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:21.780312 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:22.090482 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:22.090400 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:22.830471 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:22.830430 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:23.401479 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:23.401441 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:23.712540 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:23.712445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:23.890450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:23.890412 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:24.100001 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:24.099960 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:24.343051 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:24.343009 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:24.575920 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:24.575859 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:24.795802 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:24.792967 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:25.041586 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:25.041551 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:25.193913 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:25.193874 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:25.534413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:25.534384 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:25.726245 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:25.726212 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:26.836523 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:26.836492 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:27.022135 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:27.022100 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:27.176873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:27.176709 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:28.170443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:28.170376 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:28.545369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:28.545311 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:28.692471 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:28.692439 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:29.066816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:29.066784 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:29.640453 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:29.640419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:29.806484 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:29.806449 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:30.646742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:30.644027 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:30.994613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:30.994586 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:31.168542 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:31.168511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:31.590877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:31.590846 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:32.470474 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:32.470444 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:33.139491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:33.130431 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:33.860032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:33.859996 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:34.760676 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:34.759008 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:34.800011 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:02:34.799976 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:02:35.531205 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:35.531167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:36.350385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:36.350346 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:37.260010 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:37.259882 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:38.051977 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:38.051944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:38.803619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:38.803584 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:39.566228 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:39.565932 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:40.200843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:40.200750 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:40.898468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:40.898404 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:41.673104 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:41.672840 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:42.299961 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:42.299924 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:42.950032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:42.949983 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:43.781791 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:43.773376 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:44.437811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:44.437757 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:45.088310 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:45.088183 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:45.944513 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:45.944483 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:46.087827 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:46.087736 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:46.862802 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:46.862739 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:47.720059 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:47.719970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:48.463709 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:48.463661 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:49.158743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:49.139585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:49.807410 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:49.797906 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:50.659759 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:50.657516 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:51.516627 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:51.516593 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:51.663628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:51.663575 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:52.330873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:52.330839 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:53.114031 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:53.113984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:53.845397 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:53.845360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:54.119994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:54.119898 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:54.622838 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:54.622779 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:54.764906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:54.764862 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:54.948846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:54.948758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:55.115890 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:55.115851 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:55.575544 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:55.575510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:55.717450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:55.717413 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:56.179612 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:56.179560 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:56.964779 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:56.950312 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:57.662677 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:57.662644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:58.487339 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:58.487307 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:59.200658 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:59.200623 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:02:59.827457 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:02:59.826198 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:00.560422 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:00.560353 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:01.247291 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:01.247262 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:01.951382 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:01.951337 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:02.665014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:02.664970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:03.365396 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:03.365361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:04.116397 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:04.116355 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:04.790838 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:03:04.790801 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:03:04.852750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:04.841278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:05.494497 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:05.488003 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:06.261017 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:06.260980 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:06.396617 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:06.396580 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:07.040583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:07.040524 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:07.860537 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:07.860461 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:08.540643 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:08.540603 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:09.286542 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:09.286505 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:09.898140 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:09.898107 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:10.559309 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:10.559219 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:11.385727 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:11.385662 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:11.540897 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:11.540849 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:12.041191 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:12.041154 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:12.560053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:12.560021 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:12.699000 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:12.698967 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:12.889995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:12.889914 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:13.030387 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:13.030355 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:13.236177 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:13.236094 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:13.394112 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:13.394078 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:13.545910 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:13.545876 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:13.751759 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:13.748545 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:14.011468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:14.011436 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:14.152308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:14.152245 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:14.765093 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:14.765055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:15.472739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:15.471632 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:16.091971 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:16.090845 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:16.775998 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:16.775953 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:17.452481 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:17.452404 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:18.094098 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:18.094064 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:18.780380 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:18.780340 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:19.404775 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:19.404742 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:20.066899 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:20.066847 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:20.694517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:20.691194 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:21.252393 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:21.252357 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:21.409681 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:21.409648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:21.614187 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:21.614083 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:21.894596 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:21.894524 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:22.271141 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:22.270207 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:22.954202 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:22.954158 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:23.671107 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:23.671064 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:24.334968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:24.334927 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:24.973360 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:24.973323 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:25.774406 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:25.774259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:26.544430 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:26.544396 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:27.447526 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:27.447490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:28.253630 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:28.253596 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:28.731459 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:28.731350 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:28.881305 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:28.881278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:29.034155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:29.034119 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:29.662427 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:29.661329 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:30.371267 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:30.371235 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:31.032384 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:31.032335 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:31.656833 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:31.656780 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:32.406442 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:32.403110 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:33.070769 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:33.062443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:33.827545 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:33.827511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:34.544070 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:34.544035 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:34.791027 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:03:34.790993 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:03:35.273305 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:35.273262 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:35.438487 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:35.438459 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:35.808350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:35.808315 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:36.119908 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:36.119814 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:36.620783 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:36.620737 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:36.796498 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:36.796445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:37.556306 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:37.556231 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:38.252764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:38.252730 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:38.895805 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:38.895767 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:39.625080 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:39.625047 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:40.352161 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:40.351930 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:41.036565 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:41.036532 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:41.771775 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:41.765682 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:42.418683 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:42.418641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:43.075219 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:43.075147 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:43.819081 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:43.818808 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:44.454841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:44.454803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:45.062616 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:45.062505 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:45.793069 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:45.793034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:46.465055 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:46.465009 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:47.298821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:47.298767 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:47.979706 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:47.979672 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:48.762430 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:48.762374 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:49.429096 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:49.429013 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:50.167383 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:50.167338 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:51.178092 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:51.178056 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:51.605561 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:51.605534 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:52.029976 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:52.029939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:52.775314 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:52.772606 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:53.455642 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:53.455329 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:54.203355 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:54.199192 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:54.925767 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:54.925732 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:55.390029 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:55.389981 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:55.550299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:55.550259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:56.140741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:56.140681 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:57.037511 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:57.037479 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:57.736773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:57.736698 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:58.462625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:58.462590 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:58.621191 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:58.621125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:58.947233 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:58.947198 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj"] Oct 31 02:03:58.962642 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:58.962607 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:58.989412 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:58.989363 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj"] Oct 31 02:03:59.031821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.022355 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"kube-root-ca.crt\"" Oct 31 02:03:59.031821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.022459 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"openshift-service-ca.crt\"" Oct 31 02:03:59.031821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.022548 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"routing-overlay\"" Oct 31 02:03:59.031821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.022607 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0b6abf\"" Oct 31 02:03:59.031821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.023180 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-09706649fc65\"" Oct 31 02:03:59.031821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.023290 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-e2e--c450cd8d-dockercfg-2j4lf\"" Oct 31 02:03:59.031821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.023352 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-a62b50775dea\"" Oct 31 02:03:59.066827 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.066787 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn54v\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-kube-api-access-bn54v\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.083907 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.083877 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.084073 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.084005 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.084073 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.084023 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.185937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.185896 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn54v\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-kube-api-access-bn54v\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.186174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.186058 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.187219 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.186184 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.187219 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.186212 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.187219 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.186958 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.188231 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.188199 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.195361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.195332 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.207813 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.207706 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn54v\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-kube-api-access-bn54v\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.360459 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.360430 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.572405 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.572318 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:03:59.651537 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.651495 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj"] Oct 31 02:03:59.652969 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:03:59.652941 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd577f45b_ae56_4f5a_a462_16e2b0377b01.slice/crio-04691d53d0d77bfe25b152f2de46d9b5807b20ecd640db58dcbe6f2db68dcdba WatchSource:0}: Error finding container 04691d53d0d77bfe25b152f2de46d9b5807b20ecd640db58dcbe6f2db68dcdba: Status 404 returned error can't find the container with id 04691d53d0d77bfe25b152f2de46d9b5807b20ecd640db58dcbe6f2db68dcdba Oct 31 02:03:59.798629 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.798593 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" event={"ID":"d577f45b-ae56-4f5a-a462-16e2b0377b01","Type":"ContainerStarted","Data":"687ec5fb54e13a055857a846413bae21abbd98d9bac82058b163485cdbc4c3d4"} Oct 31 02:03:59.798856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.798645 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" event={"ID":"d577f45b-ae56-4f5a-a462-16e2b0377b01","Type":"ContainerStarted","Data":"04691d53d0d77bfe25b152f2de46d9b5807b20ecd640db58dcbe6f2db68dcdba"} Oct 31 02:03:59.798856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.798673 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:03:59.817132 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:03:59.817083 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" podStartSLOduration=1.817069313 podStartE2EDuration="1.817069313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:03:59.815810096 +0000 UTC m=+2640.196055340" watchObservedRunningTime="2026-10-31 02:03:59.817069313 +0000 UTC m=+2640.197314537" Oct 31 02:04:00.179693 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:00.179659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:00.870778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:00.869908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:01.533999 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:01.533964 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:02.135824 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:02.132317 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:02.871319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:02.870653 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:03.663151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:03.661920 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:04.290747 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:04.283497 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:04.731428 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:04.731334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:04.792400 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:04:04.792358 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:04:04.957585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:04.957555 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:05.654812 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:05.654776 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:06.348626 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:06.348593 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:07.073500 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:07.073463 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:07.753665 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:07.753625 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:08.501911 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:08.501863 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:09.184736 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:09.181625 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:09.777669 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:09.777631 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:09.931101 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:09.931062 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:10.641739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:10.639586 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:10.809969 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:10.809942 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:04:11.279951 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:11.279915 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:11.420200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:11.420164 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:12.080870 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:12.080819 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:12.781095 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:12.781037 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:13.346697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:13.346654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:14.043044 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:14.043012 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:14.685894 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:14.684254 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:15.292346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:15.292259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:15.942826 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:15.942794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:16.690740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:16.690638 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:17.171838 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:17.171783 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:17.316326 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:17.316290 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:17.476084 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:17.475993 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:18.085593 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:18.083296 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:18.593760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:18.593689 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:18.810075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:18.810038 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:19.485740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:19.478740 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:20.148364 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:20.148325 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:20.867032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:20.866959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:21.567413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:21.567365 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:22.269694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:22.269647 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:22.475034 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:22.474987 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:22.659596 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:22.659508 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:23.500415 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:23.500371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:24.149132 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:24.149103 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:24.294812 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:24.294770 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:24.570862 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:24.570823 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:24.723070 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:24.723035 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:25.408382 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:25.408337 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:25.582045 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:25.582005 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:26.286841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:26.286803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:26.964053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:26.963994 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:27.584439 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:27.584378 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:28.328039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:28.328007 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:29.240322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:29.240281 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:29.847048 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:29.847017 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:30.567814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:30.567779 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:30.724778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:30.724695 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:30.890045 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:30.889947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:31.932533 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:31.932479 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:32.182607 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:32.182523 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:32.345097 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:32.338554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:33.194631 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:33.194597 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:33.365311 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:33.365227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:34.339195 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:34.339162 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:34.514204 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:34.514166 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:34.794838 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:04:34.794799 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:04:35.369347 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:35.369314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:35.639796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:35.639672 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:36.445957 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:36.445924 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:37.095353 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:37.095283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:37.782348 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:37.781630 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:38.461041 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:38.460998 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:39.182525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:39.182467 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:39.863726 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:39.863672 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:40.599585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:40.597334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:41.463089 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:41.463043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:41.622285 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:41.622251 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:42.384579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:42.384541 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:43.082489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:43.082454 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:43.855786 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:43.855748 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:44.471120 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:44.471061 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:45.144575 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:45.144490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:45.755048 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:45.753959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:46.430088 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:46.430057 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:46.977958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:46.977916 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:47.134539 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:47.134504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:47.428282 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:47.428249 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:48.372821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:48.371149 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:49.004742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:49.002659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:49.669733 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:49.669690 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:50.288709 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:50.284876 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:50.906014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:50.905975 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:51.554946 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:51.554916 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:51.709950 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:51.709920 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:52.248741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:52.244671 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:53.096122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:53.096076 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:53.952673 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:53.952636 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:54.661140 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:54.659975 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:54.877086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:54.877033 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:55.039754 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:55.036882 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:55.523621 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:55.523589 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:55.676740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:55.671344 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:56.381478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:56.376334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:57.217741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:57.199343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:58.029620 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:58.029587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:58.743790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:58.743756 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:58.878405 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:58.878368 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:59.509113 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:59.509080 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:04:59.649278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:04:59.649214 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:00.325404 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:00.325371 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:05:00.330579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:00.330560 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:05:00.337035 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:00.337014 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:05:00.347623 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:00.347589 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:00.381363 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:00.381338 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:05:01.049738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:01.046870 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:01.838281 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:01.838245 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:02.368081 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:02.368043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:02.525991 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:02.525762 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:02.671160 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:02.671081 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:03.186254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:03.186214 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:03.875740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:03.875675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:04.648124 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:04.648085 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:04.791082 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:05:04.791047 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:05:05.067869 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:05.067832 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj"] Oct 31 02:05:05.295362 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:05.295326 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:05.460264 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:05.460183 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:05.791517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:05.791479 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:06.049034 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:06.048924 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:06.394604 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:06.394525 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:06.534468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:06.534440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:06.669556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:06.669479 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:06.826037 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:06.825964 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:07.547669 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:07.547636 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:08.150149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:08.150112 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:09.000742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:08.986397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:09.691247 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:09.691217 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:09.825847 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:09.825792 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:10.068450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.068391 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" podUID="d577f45b-ae56-4f5a-a462-16e2b0377b01" containerName="payload-processing" containerID="cri-o://687ec5fb54e13a055857a846413bae21abbd98d9bac82058b163485cdbc4c3d4" gracePeriod=25 Oct 31 02:05:10.110007 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.109985 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:10.323851 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.323769 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:10.591974 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.591776 5090 generic.go:356] "Generic (PLEG): container finished" podID="d577f45b-ae56-4f5a-a462-16e2b0377b01" containerID="687ec5fb54e13a055857a846413bae21abbd98d9bac82058b163485cdbc4c3d4" exitCode=0 Oct 31 02:05:10.591974 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.591852 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" event={"ID":"d577f45b-ae56-4f5a-a462-16e2b0377b01","Type":"ContainerDied","Data":"687ec5fb54e13a055857a846413bae21abbd98d9bac82058b163485cdbc4c3d4"} Oct 31 02:05:10.706209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.706174 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:05:10.805303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.803473 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-provider-credentials\") pod \"d577f45b-ae56-4f5a-a462-16e2b0377b01\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " Oct 31 02:05:10.805303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.803555 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-plugins-config-volume\") pod \"d577f45b-ae56-4f5a-a462-16e2b0377b01\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " Oct 31 02:05:10.805303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.803743 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-kube-api-access-bn54v\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-kube-api-access-bn54v\") pod \"d577f45b-ae56-4f5a-a462-16e2b0377b01\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " Oct 31 02:05:10.805303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.803880 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-routing-overlay-volume\") pod \"d577f45b-ae56-4f5a-a462-16e2b0377b01\" (UID: \"d577f45b-ae56-4f5a-a462-16e2b0377b01\") " Oct 31 02:05:10.807526 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.807500 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-plugins-config-volume" pod "d577f45b-ae56-4f5a-a462-16e2b0377b01" (UID: "d577f45b-ae56-4f5a-a462-16e2b0377b01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:10.810460 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.808217 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-provider-credentials" pod "d577f45b-ae56-4f5a-a462-16e2b0377b01" (UID: "d577f45b-ae56-4f5a-a462-16e2b0377b01"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:10.810852 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.810771 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-routing-overlay-volume" pod "d577f45b-ae56-4f5a-a462-16e2b0377b01" (UID: "d577f45b-ae56-4f5a-a462-16e2b0377b01"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:10.828808 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.828770 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-kube-api-access-bn54v" pod "d577f45b-ae56-4f5a-a462-16e2b0377b01" (UID: "d577f45b-ae56-4f5a-a462-16e2b0377b01"). InnerVolumeSpecName "kube-api-access-bn54v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:10.836367 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.836340 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:10.907110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.907077 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn54v\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-kube-api-access-bn54v\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:10.907110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.907108 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:10.907361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.907117 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d577f45b-ae56-4f5a-a462-16e2b0377b01-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:10.907361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:10.907126 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d577f45b-ae56-4f5a-a462-16e2b0377b01-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:11.076141 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.076107 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:11.085790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.085742 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm"] Oct 31 02:05:11.087049 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.087020 5090 cpu_manager.go:414] "containerMap: removing container" podUID="d577f45b-ae56-4f5a-a462-16e2b0377b01" containerName="payload-processing" Oct 31 02:05:11.087049 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.087047 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d577f45b-ae56-4f5a-a462-16e2b0377b01" containerName="payload-processing" Oct 31 02:05:11.087292 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.087279 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="d577f45b-ae56-4f5a-a462-16e2b0377b01" containerName="payload-processing" Oct 31 02:05:11.091625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.091601 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.103582 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.103548 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm"] Oct 31 02:05:11.117382 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.117354 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"e2e-cmp-legacy-03f7cb-provider-cred\"" Oct 31 02:05:11.117573 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.117551 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"kube-root-ca.crt\"" Oct 31 02:05:11.117573 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.117570 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"openshift-service-ca.crt\"" Oct 31 02:05:11.117781 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.117557 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-20538a\"" Oct 31 02:05:11.117979 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.117945 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"routing-overlay\"" Oct 31 02:05:11.213368 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.213261 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.213599 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.213578 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.213678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.213658 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.213851 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.213823 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mhzb\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-kube-api-access-6mhzb\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.332764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.325534 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.332764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.325803 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.332764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.325880 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.332764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.325943 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mhzb\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-kube-api-access-6mhzb\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.332764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.327059 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.332764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.328158 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.352325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.352286 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.371107 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.361454 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mhzb\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-kube-api-access-6mhzb\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.414853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.414467 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:11.604863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.604831 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" event={"ID":"d577f45b-ae56-4f5a-a462-16e2b0377b01","Type":"ContainerDied","Data":"04691d53d0d77bfe25b152f2de46d9b5807b20ecd640db58dcbe6f2db68dcdba"} Oct 31 02:05:11.604863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.604859 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj" Oct 31 02:05:11.605143 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.604906 5090 scope.go:122] "RemoveContainer" containerID="687ec5fb54e13a055857a846413bae21abbd98d9bac82058b163485cdbc4c3d4" Oct 31 02:05:11.648350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.648314 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj"] Oct 31 02:05:11.658045 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.657999 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-678972dkj"] Oct 31 02:05:11.733630 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.733590 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm"] Oct 31 02:05:11.734499 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:05:11.734461 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a9f323_9e08_4685_adca_fe27a5664068.slice/crio-d479e4f48c83fb65670d74d6531435319bf8c23af0689ef1be4cf058791efd5a WatchSource:0}: Error finding container d479e4f48c83fb65670d74d6531435319bf8c23af0689ef1be4cf058791efd5a: Status 404 returned error can't find the container with id d479e4f48c83fb65670d74d6531435319bf8c23af0689ef1be4cf058791efd5a Oct 31 02:05:11.752172 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.752115 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d577f45b-ae56-4f5a-a462-16e2b0377b01" path="/var/lib/kubelet/pods/d577f45b-ae56-4f5a-a462-16e2b0377b01/volumes" Oct 31 02:05:11.936040 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:11.934539 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:12.573780 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:12.573692 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:12.616093 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:12.616052 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" event={"ID":"e7a9f323-9e08-4685-adca-fe27a5664068","Type":"ContainerStarted","Data":"2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1"} Oct 31 02:05:12.616299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:12.616114 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" event={"ID":"e7a9f323-9e08-4685-adca-fe27a5664068","Type":"ContainerStarted","Data":"d479e4f48c83fb65670d74d6531435319bf8c23af0689ef1be4cf058791efd5a"} Oct 31 02:05:12.616299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:12.616148 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:12.640849 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:12.640796 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" podStartSLOduration=1.640777093 podStartE2EDuration="1.640777093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:12.63551225 +0000 UTC m=+2713.015757486" watchObservedRunningTime="2026-10-31 02:05:12.640777093 +0000 UTC m=+2713.021022316" Oct 31 02:05:12.982795 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:12.982202 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:13.130918 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:13.130883 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:13.354529 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:13.354494 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:13.552154 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:13.552087 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:13.707642 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:13.707540 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:13.860564 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:13.860511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:14.003572 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:14.003545 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:14.225411 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:14.225370 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:14.409993 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:14.409915 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:14.568751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:14.562188 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:14.711796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:14.709867 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:15.127855 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:15.127809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:15.417328 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:15.417228 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:15.583641 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:15.583596 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:15.755840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:15.755811 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:15.934739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:15.933772 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:16.202643 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:16.194869 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:16.537969 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:16.537123 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:16.742388 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:16.727969 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:16.886978 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:16.886894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:17.217184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:17.217088 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:17.414877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:17.414842 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:18.339745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:18.328426 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:19.060216 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:19.060179 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:19.374684 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:19.374587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:19.555027 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:19.554898 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:20.785531 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:20.785504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:20.981448 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:20.981409 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:21.129798 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:21.129708 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:21.318751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:21.318248 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:21.471023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:21.470943 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:21.747251 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:21.747220 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:21.925796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:21.925443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:22.393079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:22.393038 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:22.648876 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:22.648754 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:23.224685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:23.224645 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:23.426987 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:23.426954 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:23.586536 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:23.586492 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:23.628085 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:23.628056 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:05:24.755885 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:24.755857 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:24.952967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:24.952935 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:25.109853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:25.109696 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:25.837148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:25.837112 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:25.995678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:25.995642 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:26.933396 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:26.931438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:27.640598 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:27.640548 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:28.345258 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:28.345226 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:29.036646 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:29.035980 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:29.671162 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:29.671129 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:30.363112 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:30.363067 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:31.027991 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:31.027937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:31.681172 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:31.681131 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:32.372273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:32.372241 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:33.158286 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:33.158247 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:33.757638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:33.757609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:34.471150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:34.471113 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:34.791627 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:05:34.791591 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:05:35.100808 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:35.096459 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:35.722781 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:35.722753 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:36.438697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:36.438654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:37.173048 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:37.173016 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:37.646104 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:37.646065 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:37.793088 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:37.793051 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:38.213085 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:38.213049 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:39.151327 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:39.151240 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:39.864757 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:39.859140 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:40.467384 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:40.467347 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:41.155671 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:41.155639 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:41.845079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:41.845017 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:42.509816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.509758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:42.683985 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.683936 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q"] Oct 31 02:05:42.684308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.684259 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" podUID="ae11a79a-08da-477e-8fc4-a8593d626487" containerName="maas-api" containerID="cri-o://ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417" gracePeriod=30 Oct 31 02:05:42.699350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.699243 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8"] Oct 31 02:05:42.699576 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.699528 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" containerID="cri-o://1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6" gracePeriod=30 Oct 31 02:05:42.714492 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.712263 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd"] Oct 31 02:05:42.714492 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.712840 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" containerID="cri-o://4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555" gracePeriod=30 Oct 31 02:05:42.720670 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.720360 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr"] Oct 31 02:05:42.720670 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.720532 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" containerID="cri-o://4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4" gracePeriod=30 Oct 31 02:05:42.730517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.730491 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r"] Oct 31 02:05:42.731764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:42.731695 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" containerID="cri-o://ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e" gracePeriod=30 Oct 31 02:05:43.004440 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.004415 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:05:43.183890 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.183804 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae11a79a-08da-477e-8fc4-a8593d626487-kube-api-access-z5wzk\" (UniqueName: \"kubernetes.io/projected/ae11a79a-08da-477e-8fc4-a8593d626487-kube-api-access-z5wzk\") pod \"ae11a79a-08da-477e-8fc4-a8593d626487\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " Oct 31 02:05:43.183890 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.183839 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-metrics-certs\") pod \"ae11a79a-08da-477e-8fc4-a8593d626487\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " Oct 31 02:05:43.184129 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.183989 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls\") pod \"ae11a79a-08da-477e-8fc4-a8593d626487\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " Oct 31 02:05:43.184129 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.184125 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae11a79a-08da-477e-8fc4-a8593d626487-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae11a79a-08da-477e-8fc4-a8593d626487-openshift-service-ca\") pod \"ae11a79a-08da-477e-8fc4-a8593d626487\" (UID: \"ae11a79a-08da-477e-8fc4-a8593d626487\") " Oct 31 02:05:43.184560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.184517 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae11a79a-08da-477e-8fc4-a8593d626487-openshift-service-ca" pod "ae11a79a-08da-477e-8fc4-a8593d626487" (UID: "ae11a79a-08da-477e-8fc4-a8593d626487"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:43.184781 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.184760 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae11a79a-08da-477e-8fc4-a8593d626487-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:43.187078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.187047 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls" pod "ae11a79a-08da-477e-8fc4-a8593d626487" (UID: "ae11a79a-08da-477e-8fc4-a8593d626487"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:05:43.187214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.187190 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae11a79a-08da-477e-8fc4-a8593d626487-kube-api-access-z5wzk" pod "ae11a79a-08da-477e-8fc4-a8593d626487" (UID: "ae11a79a-08da-477e-8fc4-a8593d626487"). InnerVolumeSpecName "kube-api-access-z5wzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:43.187434 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.187410 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-metrics-certs" pod "ae11a79a-08da-477e-8fc4-a8593d626487" (UID: "ae11a79a-08da-477e-8fc4-a8593d626487"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:05:43.213988 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.213959 5090 generic.go:356] "Generic (PLEG): container finished" podID="ae11a79a-08da-477e-8fc4-a8593d626487" containerID="ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417" exitCode=0 Oct 31 02:05:43.214116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.214008 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" Oct 31 02:05:43.214116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.214048 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" event={"ID":"ae11a79a-08da-477e-8fc4-a8593d626487","Type":"ContainerDied","Data":"ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417"} Oct 31 02:05:43.214116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.214108 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q" event={"ID":"ae11a79a-08da-477e-8fc4-a8593d626487","Type":"ContainerDied","Data":"847ed9bc0eb141b93a81b0618a7904a43ec491b98efc35a016e2411bc38561b3"} Oct 31 02:05:43.214261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.214132 5090 scope.go:122] "RemoveContainer" containerID="ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417" Oct 31 02:05:43.227679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.227658 5090 scope.go:122] "RemoveContainer" containerID="ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417" Oct 31 02:05:43.228013 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:05:43.227994 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417\": container with ID starting with ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417 not found: ID does not exist" containerID="ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417" Oct 31 02:05:43.228084 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.228022 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417"} err="failed to get container status \"ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417\": rpc error: code = NotFound desc = could not find container \"ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417\": container with ID starting with ff8d3db43f8b21ab40ca0d8f941a46724abfce8e08c6796af5f92ea8d24fb417 not found: ID does not exist" Oct 31 02:05:43.241919 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.241794 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q"] Oct 31 02:05:43.246272 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.246245 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91b6b1-55f755764d-qvh9q"] Oct 31 02:05:43.286273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.286245 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wzk\" (UniqueName: \"kubernetes.io/projected/ae11a79a-08da-477e-8fc4-a8593d626487-kube-api-access-z5wzk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:43.286386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.286274 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:43.286386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.286290 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae11a79a-08da-477e-8fc4-a8593d626487-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:43.451883 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.446586 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:43.749760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:43.749725 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae11a79a-08da-477e-8fc4-a8593d626487" path="/var/lib/kubelet/pods/ae11a79a-08da-477e-8fc4-a8593d626487/volumes" Oct 31 02:05:44.269075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:44.269038 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:05:44.269547 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:44.269114 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:05:44.484743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:44.483284 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:44.732223 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:44.732128 5090 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:05:44.732223 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:44.732194 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:05:45.251742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:45.246006 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:46.167514 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:46.167458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:46.612229 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:46.612180 5090 patch_prober.go:29] interesting pod/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:05:46.612433 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:46.612250 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:05:47.094741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:47.090794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:47.980573 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:47.980446 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:48.096891 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.096856 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:05:48.194146 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.194114 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:05:48.199580 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.199555 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:05:48.204256 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.204231 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:05:48.216974 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.216950 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-data\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217126 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63a9e50-e475-408e-887a-0d49721bd7e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63a9e50-e475-408e-887a-0d49721bd7e4-istiod-ca-cert\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217153 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-certs\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217207 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-credential-socket\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217245 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-podinfo\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217267 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-data" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.217667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217378 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-token\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-token\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217433 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-socket\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217476 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-kube-api-access-69zq6\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-kube-api-access-69zq6\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.217667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217492 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-certs" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.217667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217633 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63a9e50-e475-408e-887a-0d49721bd7e4-istiod-ca-cert" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:48.219465 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217771 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-socket" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.219609 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217802 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-credential-socket" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.219609 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.217706 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-envoy\") pod \"d63a9e50-e475-408e-887a-0d49721bd7e4\" (UID: \"d63a9e50-e475-408e-887a-0d49721bd7e4\") " Oct 31 02:05:48.220397 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.220361 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-podinfo" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:05:48.220525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.220500 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-kube-api-access-69zq6" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "kube-api-access-69zq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.220895 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.220873 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-envoy" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.221075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221055 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221078 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d63a9e50-e475-408e-887a-0d49721bd7e4-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221088 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221097 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221105 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221115 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221124 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69zq6\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-kube-api-access-69zq6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221134 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.221617 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.221598 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-token" pod "d63a9e50-e475-408e-887a-0d49721bd7e4" (UID: "d63a9e50-e475-408e-887a-0d49721bd7e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.281616 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.281585 5090 generic.go:356] "Generic (PLEG): container finished" podID="5a369468-1514-41de-9a9e-4fbc627b0095" containerID="4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555" exitCode=0 Oct 31 02:05:48.282169 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.282112 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" event={"ID":"5a369468-1514-41de-9a9e-4fbc627b0095","Type":"ContainerDied","Data":"4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555"} Oct 31 02:05:48.282443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.282421 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" event={"ID":"5a369468-1514-41de-9a9e-4fbc627b0095","Type":"ContainerDied","Data":"b6d2c51be4412ca2fdba94e2cd50a0d8c1f7c98cc8a7b6d4463f717809afe5a9"} Oct 31 02:05:48.282443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.282463 5090 scope.go:122] "RemoveContainer" containerID="4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555" Oct 31 02:05:48.285164 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.282613 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd" Oct 31 02:05:48.292345 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.292321 5090 generic.go:356] "Generic (PLEG): container finished" podID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerID="1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6" exitCode=0 Oct 31 02:05:48.292477 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.292463 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" Oct 31 02:05:48.298306 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.298280 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" event={"ID":"d63a9e50-e475-408e-887a-0d49721bd7e4","Type":"ContainerDied","Data":"1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6"} Oct 31 02:05:48.298520 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.298463 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8" event={"ID":"d63a9e50-e475-408e-887a-0d49721bd7e4","Type":"ContainerDied","Data":"a9299bb68cfe1edd1e6192c7ed93e46fc739ae3bf4f208e76da9f0a78198daca"} Oct 31 02:05:48.299933 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.299909 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq"] Oct 31 02:05:48.300510 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.300490 5090 generic.go:356] "Generic (PLEG): container finished" podID="482267d2-2956-4473-af8a-112c174a7b35" containerID="4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4" exitCode=0 Oct 31 02:05:48.300688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.300669 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" Oct 31 02:05:48.301014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.300996 5090 cpu_manager.go:414] "containerMap: removing container" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" Oct 31 02:05:48.301145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301017 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" Oct 31 02:05:48.301145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301033 5090 cpu_manager.go:414] "containerMap: removing container" podUID="ae11a79a-08da-477e-8fc4-a8593d626487" containerName="maas-api" Oct 31 02:05:48.301145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301040 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae11a79a-08da-477e-8fc4-a8593d626487" containerName="maas-api" Oct 31 02:05:48.301145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301070 5090 cpu_manager.go:414] "containerMap: removing container" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" Oct 31 02:05:48.301145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301078 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" Oct 31 02:05:48.301145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301130 5090 cpu_manager.go:414] "containerMap: removing container" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" Oct 31 02:05:48.301145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301137 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" Oct 31 02:05:48.301488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301158 5090 cpu_manager.go:414] "containerMap: removing container" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" Oct 31 02:05:48.301488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301165 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" Oct 31 02:05:48.301488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301345 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" containerName="istio-proxy" Oct 31 02:05:48.301488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301361 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" containerName="istio-proxy" Oct 31 02:05:48.301488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301374 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="482267d2-2956-4473-af8a-112c174a7b35" containerName="istio-proxy" Oct 31 02:05:48.301488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301384 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae11a79a-08da-477e-8fc4-a8593d626487" containerName="maas-api" Oct 31 02:05:48.301488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.301394 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerName="istio-proxy" Oct 31 02:05:48.305120 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.305067 5090 generic.go:356] "Generic (PLEG): container finished" podID="b130cabf-4761-43c0-bfce-7b34b0aceba2" containerID="ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e" exitCode=0 Oct 31 02:05:48.305196 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.305146 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" Oct 31 02:05:48.314139 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.314102 5090 scope.go:122] "RemoveContainer" containerID="4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555" Oct 31 02:05:48.314477 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:05:48.314457 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555\": container with ID starting with 4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555 not found: ID does not exist" containerID="4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555" Oct 31 02:05:48.314569 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.314487 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555"} err="failed to get container status \"4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555\": rpc error: code = NotFound desc = could not find container \"4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555\": container with ID starting with 4e31f36fc35efbe1d1bd657f25160e68d5a138679e77c1e0190b3aa58f91e555 not found: ID does not exist" Oct 31 02:05:48.314569 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.314505 5090 scope.go:122] "RemoveContainer" containerID="1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6" Oct 31 02:05:48.315099 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.315080 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq"] Oct 31 02:05:48.315200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.315123 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" event={"ID":"482267d2-2956-4473-af8a-112c174a7b35","Type":"ContainerDied","Data":"4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4"} Oct 31 02:05:48.315200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.315149 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr" event={"ID":"482267d2-2956-4473-af8a-112c174a7b35","Type":"ContainerDied","Data":"5dd3cb7ac4cf468a39cf18b566376e852020f081da002b6c8e2031a9532bb6d0"} Oct 31 02:05:48.315200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.315167 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" event={"ID":"b130cabf-4761-43c0-bfce-7b34b0aceba2","Type":"ContainerDied","Data":"ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e"} Oct 31 02:05:48.315200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.315193 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r" event={"ID":"b130cabf-4761-43c0-bfce-7b34b0aceba2","Type":"ContainerDied","Data":"41b9887c95e95d8ad30044849487af0ac2d0148e8acf0bdf9dc825e3bba48db1"} Oct 31 02:05:48.315437 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.315270 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.322583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.322562 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-data\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.322754 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.322736 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-podinfo\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.322859 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.322844 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b130cabf-4761-43c0-bfce-7b34b0aceba2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130cabf-4761-43c0-bfce-7b34b0aceba2-istiod-ca-cert\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.322991 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.322975 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-credential-socket\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.323100 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.323070 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-data" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.323207 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.323191 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-kube-api-access-jdrn4\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-kube-api-access-jdrn4\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.323328 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.323310 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-certs\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.323464 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.323438 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b130cabf-4761-43c0-bfce-7b34b0aceba2-istiod-ca-cert" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:48.323574 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.323542 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-socket\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.323689 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.323673 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-envoy\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.324237 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.324220 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-envoy\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.324602 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.324583 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-credential-socket\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.325995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.325003 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-istio-token\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-istio-token\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.325995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.323826 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-certs" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.325995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.324146 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-socket" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.325995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.324844 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-credential-socket" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.325995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.324935 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-credential-socket" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.325995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.325958 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482267d2-2956-4473-af8a-112c174a7b35-istiod-ca-cert" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:48.326792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.325114 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482267d2-2956-4473-af8a-112c174a7b35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482267d2-2956-4473-af8a-112c174a7b35-istiod-ca-cert\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.326905 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.326817 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-data\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.326905 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.326848 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-token\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-token\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.327323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.326924 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-istio-token\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-istio-token\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.327323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327204 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-data" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.327553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327375 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a369468-1514-41de-9a9e-4fbc627b0095-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a369468-1514-41de-9a9e-4fbc627b0095-istiod-ca-cert\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.327553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327523 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-socket\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.327761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327556 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-credential-socket\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.327761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327612 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-kube-api-access-6wlxb\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-kube-api-access-6wlxb\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.327761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327661 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-data\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.327761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327708 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-certs\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.328319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327760 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-kube-api-access-nzqdh\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-kube-api-access-nzqdh\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.328319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327822 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/482267d2-2956-4473-af8a-112c174a7b35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482267d2-2956-4473-af8a-112c174a7b35-istio-podinfo\") pod \"482267d2-2956-4473-af8a-112c174a7b35\" (UID: \"482267d2-2956-4473-af8a-112c174a7b35\") " Oct 31 02:05:48.328319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327861 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a369468-1514-41de-9a9e-4fbc627b0095-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a369468-1514-41de-9a9e-4fbc627b0095-istio-podinfo\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.328319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327901 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-certs\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.328319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327945 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-socket\") pod \"5a369468-1514-41de-9a9e-4fbc627b0095\" (UID: \"5a369468-1514-41de-9a9e-4fbc627b0095\") " Oct 31 02:05:48.328319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.327987 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-envoy\") pod \"b130cabf-4761-43c0-bfce-7b34b0aceba2\" (UID: \"b130cabf-4761-43c0-bfce-7b34b0aceba2\") " Oct 31 02:05:48.328319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.328166 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-data" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.330276 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.330252 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a369468-1514-41de-9a9e-4fbc627b0095-istiod-ca-cert" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:05:48.330541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.330495 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-socket" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.331463 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.331443 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-certs" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.332276 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332252 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d63a9e50-e475-408e-887a-0d49721bd7e4-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332281 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332299 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482267d2-2956-4473-af8a-112c174a7b35-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332313 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332326 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a369468-1514-41de-9a9e-4fbc627b0095-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332340 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332362 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332379 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332394 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332408 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130cabf-4761-43c0-bfce-7b34b0aceba2-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332421 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332434 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.332458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332448 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.333672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.332891 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-credential-socket" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.333672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.333086 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-token" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.333672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.333230 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-istio-token" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.333672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.333359 5090 scope.go:122] "RemoveContainer" containerID="1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6" Oct 31 02:05:48.333993 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:05:48.333760 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6\": container with ID starting with 1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6 not found: ID does not exist" containerID="1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6" Oct 31 02:05:48.333993 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.333808 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6"} err="failed to get container status \"1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6\": rpc error: code = NotFound desc = could not find container \"1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6\": container with ID starting with 1e2e3b1572914af8aa8119836e1a7d90305b01273ec30006982e89df5b1ccbf6 not found: ID does not exist" Oct 31 02:05:48.333993 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.333834 5090 scope.go:122] "RemoveContainer" containerID="4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4" Oct 31 02:05:48.339100 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.339069 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-kube-api-access-nzqdh" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "kube-api-access-nzqdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.340678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.340652 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-socket" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.340819 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.340702 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-certs" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.340925 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.340830 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8"] Oct 31 02:05:48.342237 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.342201 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-envoy" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.342347 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.342235 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-kube-api-access-6wlxb" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "kube-api-access-6wlxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.342347 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.342272 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-istio-token" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.345865 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.344230 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/482267d2-2956-4473-af8a-112c174a7b35-istio-podinfo" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:05:48.346040 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.346013 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a369468-1514-41de-9a9e-4fbc627b0095-istio-podinfo" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:05:48.349086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.349061 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-envoy" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.349290 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.349270 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-kube-api-access-jdrn4" pod "482267d2-2956-4473-af8a-112c174a7b35" (UID: "482267d2-2956-4473-af8a-112c174a7b35"). InnerVolumeSpecName "kube-api-access-jdrn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:05:48.349368 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.349303 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-envoy" pod "5a369468-1514-41de-9a9e-4fbc627b0095" (UID: "5a369468-1514-41de-9a9e-4fbc627b0095"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:05:48.350305 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.350273 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d0c6ff-openshift-default-65866c4b9-mbtr8"] Oct 31 02:05:48.352042 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.352014 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-podinfo" pod "b130cabf-4761-43c0-bfce-7b34b0aceba2" (UID: "b130cabf-4761-43c0-bfce-7b34b0aceba2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:05:48.355116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.355100 5090 scope.go:122] "RemoveContainer" containerID="4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4" Oct 31 02:05:48.355429 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:05:48.355371 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4\": container with ID starting with 4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4 not found: ID does not exist" containerID="4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4" Oct 31 02:05:48.355429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.355410 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4"} err="failed to get container status \"4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4\": rpc error: code = NotFound desc = could not find container \"4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4\": container with ID starting with 4386cd272b811da080491af5b0edb08df45fba8d7d8f19f71498fc63584e76b4 not found: ID does not exist" Oct 31 02:05:48.355591 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.355433 5090 scope.go:122] "RemoveContainer" containerID="ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e" Oct 31 02:05:48.369415 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.369388 5090 scope.go:122] "RemoveContainer" containerID="ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e" Oct 31 02:05:48.369816 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:05:48.369789 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e\": container with ID starting with ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e not found: ID does not exist" containerID="ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e" Oct 31 02:05:48.369959 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.369830 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e"} err="failed to get container status \"ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e\": rpc error: code = NotFound desc = could not find container \"ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e\": container with ID starting with ed5e0cdc834af57a7423b79788804abf2b84b14297694a0317dfdd1063743c7e not found: ID does not exist" Oct 31 02:05:48.434152 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434112 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.434152 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434156 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.434386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434185 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6w2\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-kube-api-access-rs6w2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.434386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434355 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08527e47-57a3-4a56-a70a-8681a6bce40e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.434475 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434410 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.434475 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434451 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.434557 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434478 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.434646 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434628 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434683 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434650 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wlxb\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-kube-api-access-6wlxb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434683 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434662 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqdh\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-kube-api-access-nzqdh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434683 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434671 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482267d2-2956-4473-af8a-112c174a7b35-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434683 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434682 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a369468-1514-41de-9a9e-4fbc627b0095-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434691 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434699 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434708 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434730 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434740 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdrn4\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-kube-api-access-jdrn4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434748 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a369468-1514-41de-9a9e-4fbc627b0095-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434758 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482267d2-2956-4473-af8a-112c174a7b35-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434766 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482267d2-2956-4473-af8a-112c174a7b35-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434774 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130cabf-4761-43c0-bfce-7b34b0aceba2-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.434829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.434782 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a369468-1514-41de-9a9e-4fbc627b0095-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:05:48.535606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.535522 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08527e47-57a3-4a56-a70a-8681a6bce40e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.535606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.535593 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.535889 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.535617 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.535889 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.535644 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.535889 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.535803 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.535889 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.535830 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.535889 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.535856 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6w2\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-kube-api-access-rs6w2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.536844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.536781 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.538224 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.538203 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08527e47-57a3-4a56-a70a-8681a6bce40e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.538447 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.538425 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.538491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.538425 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.538560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.538543 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.549456 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.549432 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.549685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.549668 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6w2\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-kube-api-access-rs6w2\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k8mmq\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.608773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.608701 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd"] Oct 31 02:05:48.614326 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.614288 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae5a1c-openshift-default-6dbf75d4bc-vfrdd"] Oct 31 02:05:48.627949 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.627921 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr"] Oct 31 02:05:48.632353 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.632327 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91b6b1-openshift-default-5ffb686549-gcjsr"] Oct 31 02:05:48.643650 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.643624 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r"] Oct 31 02:05:48.648793 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.648772 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-4jx7r"] Oct 31 02:05:48.652526 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.652509 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:48.857796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.857655 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:48.947525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.947494 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq"] Oct 31 02:05:48.950013 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:05:48.949979 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08527e47_57a3_4a56_a70a_8681a6bce40e.slice/crio-6a7f23d40f4dcaf6c24a197010aa0395c0d94942dd1ec31ebf96de988764210c WatchSource:0}: Error finding container 6a7f23d40f4dcaf6c24a197010aa0395c0d94942dd1ec31ebf96de988764210c: Status 404 returned error can't find the container with id 6a7f23d40f4dcaf6c24a197010aa0395c0d94942dd1ec31ebf96de988764210c Oct 31 02:05:48.953900 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.953854 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:05:48.954013 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:48.953943 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:05:49.333999 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.333958 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" event={"ID":"08527e47-57a3-4a56-a70a-8681a6bce40e","Type":"ContainerStarted","Data":"16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196"} Oct 31 02:05:49.334939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.334024 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" event={"ID":"08527e47-57a3-4a56-a70a-8681a6bce40e","Type":"ContainerStarted","Data":"6a7f23d40f4dcaf6c24a197010aa0395c0d94942dd1ec31ebf96de988764210c"} Oct 31 02:05:49.335444 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.335408 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:49.413536 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.413459 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" podStartSLOduration=7.413433282 podStartE2EDuration="7.413433282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:05:49.391493288 +0000 UTC m=+2749.771738534" watchObservedRunningTime="2026-10-31 02:05:49.413433282 +0000 UTC m=+2749.793678513" Oct 31 02:05:49.599489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.599355 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:49.747622 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.745911 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:49.840136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.840098 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482267d2-2956-4473-af8a-112c174a7b35" path="/var/lib/kubelet/pods/482267d2-2956-4473-af8a-112c174a7b35/volumes" Oct 31 02:05:49.847739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.841163 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a369468-1514-41de-9a9e-4fbc627b0095" path="/var/lib/kubelet/pods/5a369468-1514-41de-9a9e-4fbc627b0095/volumes" Oct 31 02:05:49.848799 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.848770 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b130cabf-4761-43c0-bfce-7b34b0aceba2" path="/var/lib/kubelet/pods/b130cabf-4761-43c0-bfce-7b34b0aceba2/volumes" Oct 31 02:05:49.854807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:49.849667 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63a9e50-e475-408e-887a-0d49721bd7e4" path="/var/lib/kubelet/pods/d63a9e50-e475-408e-887a-0d49721bd7e4/volumes" Oct 31 02:05:50.179052 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:50.178959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:50.321879 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:50.321834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:50.345927 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:50.345892 5090 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k8mmq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:05:50.346345 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:50.345953 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" podUID="08527e47-57a3-4a56-a70a-8681a6bce40e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:05:50.882936 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:50.882870 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:51.351070 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:51.351038 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:05:51.693223 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:51.693145 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:51.835945 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:51.835908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:52.483742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:52.483206 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:53.196977 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:53.196942 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:53.847791 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:53.846261 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:54.553403 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:54.553371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:55.221827 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:55.221790 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:55.931440 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:55.931409 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:56.578136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:56.578102 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:56.715663 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:56.715634 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:56.863527 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:56.863447 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:57.425847 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:57.425809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:58.147325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:58.140224 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:05:58.841275 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:05:58.841159 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:00.189741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:00.187543 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:00.901207 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:00.901176 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:01.465744 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:01.459555 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:02.175432 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:02.175386 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:02.963739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:02.960129 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:03.521989 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:03.521959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:03.670273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:03.670239 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:04.486252 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:04.486216 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:04.669189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:04.669158 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:04.827994 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:06:04.827958 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:06:05.370245 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:05.370203 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:05.603959 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:05.603920 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:06.394971 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:06.394933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:07.031134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:07.031096 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:07.771116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:07.771078 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:08.583565 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:08.583462 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:09.233327 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:09.233290 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:09.970695 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:09.970659 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:10.671322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:10.671280 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:11.320007 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:11.319952 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:11.987420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:11.987389 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:12.676925 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:12.676888 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:13.453740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:13.453585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:14.161273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:14.161230 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:14.931093 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:14.931058 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:15.635125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:15.635093 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:16.571817 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:16.571783 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:17.270214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:17.270180 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:17.946245 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:17.946213 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:18.673162 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:18.673108 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:19.348560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:19.348523 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:20.053365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:20.053327 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:20.767444 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:20.767400 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:21.427625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:21.427587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:22.176311 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:22.176274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:22.848878 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:22.848077 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:23.465837 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:23.465798 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:24.272751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:24.272693 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:25.016334 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:25.016298 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:25.813483 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:25.813443 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:26.743767 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:26.743735 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:27.540184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:27.540147 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:28.743744 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:28.743061 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:29.621082 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:29.621055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:30.439238 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:30.439205 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:31.299680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:31.299643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:31.952859 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:31.952786 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:32.590955 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:32.590918 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:33.249128 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:33.249092 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:33.906200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:33.906165 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:34.550786 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:34.550748 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:34.789688 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:06:34.789653 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:06:35.175947 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:35.175898 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:35.779601 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:35.779567 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:36.440836 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:36.440773 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:37.261187 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:37.261149 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:37.933625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:37.933273 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:38.569660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:38.569623 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:39.269441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:39.269399 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:39.955058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:39.950683 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:40.662221 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:40.662187 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:41.347737 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:41.347684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:42.092406 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:42.092373 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:42.742657 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:42.742554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:43.358400 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:43.358360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:44.124053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:44.124018 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:44.810273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:44.810116 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:45.438758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:45.438698 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:46.270607 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:46.270520 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:47.163447 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:47.163417 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:47.850520 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:47.850488 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:48.639856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:48.639801 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:49.358627 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:49.358563 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:50.148017 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:50.147967 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:50.363098 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:50.363047 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:50.504496 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:50.504455 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:51.225130 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:51.225068 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:51.375744 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:51.374234 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:51.740574 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:51.740543 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:52.343249 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:52.343198 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:53.065531 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:53.057951 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:53.715266 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:53.715219 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:54.480642 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:54.480604 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:55.157013 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:55.156978 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:55.841336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:55.841254 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:56.503580 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:56.503527 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:57.360452 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:57.360401 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:58.082591 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:58.082554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:58.685542 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:58.685506 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:06:59.313236 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:06:59.313199 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:00.040425 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:00.040389 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:00.566531 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:00.566494 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm"] Oct 31 02:07:00.662203 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:00.662170 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:00.806057 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:00.806016 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:00.955174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:00.955026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:01.118436 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:01.115823 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:01.408128 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:01.408099 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:01.563757 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:01.563695 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:01.721381 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:01.721294 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:01.900469 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:01.900439 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:02.054273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:02.054220 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:02.243351 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:02.243319 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:02.733744 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:02.733666 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:02.872973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:02.872936 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:03.018706 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:03.018672 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:03.644474 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:03.644440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:03.782986 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:03.782948 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:04.214673 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:04.214637 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:04.791210 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:07:04.791164 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:07:04.968907 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:04.968861 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:05.567812 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.567773 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" podUID="e7a9f323-9e08-4685-adca-fe27a5664068" containerName="payload-processing" containerID="cri-o://2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1" gracePeriod=25 Oct 31 02:07:05.733507 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.733430 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:05.901176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.901151 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:07:05.944005 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.943973 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7"] Oct 31 02:07:05.945124 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.945100 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e7a9f323-9e08-4685-adca-fe27a5664068" containerName="payload-processing" Oct 31 02:07:05.945124 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.945123 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a9f323-9e08-4685-adca-fe27a5664068" containerName="payload-processing" Oct 31 02:07:05.945366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.945352 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a9f323-9e08-4685-adca-fe27a5664068" containerName="payload-processing" Oct 31 02:07:05.950059 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.950035 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:05.956125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.956099 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"openshift-service-ca.crt\"" Oct 31 02:07:05.956244 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.956143 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-e2e--43356fff-dockercfg-qvkjn\"" Oct 31 02:07:05.956244 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.956213 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bb5c51\"" Oct 31 02:07:05.956336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.956278 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"routing-overlay\"" Oct 31 02:07:05.956380 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.956334 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"agc-cred-e107317a3c62\"" Oct 31 02:07:05.956421 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.956410 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"kube-root-ca.crt\"" Oct 31 02:07:05.956485 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.956472 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"agc-cred-76f1cc2c68de\"" Oct 31 02:07:05.957504 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:05.957483 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7"] Oct 31 02:07:06.038883 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.038851 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-kube-api-access-6mhzb\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-kube-api-access-6mhzb\") pod \"e7a9f323-9e08-4685-adca-fe27a5664068\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " Oct 31 02:07:06.039060 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.038911 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-plugins-config-volume\") pod \"e7a9f323-9e08-4685-adca-fe27a5664068\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " Oct 31 02:07:06.039060 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039009 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-routing-overlay-volume\") pod \"e7a9f323-9e08-4685-adca-fe27a5664068\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " Oct 31 02:07:06.039060 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039050 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-provider-credentials\") pod \"e7a9f323-9e08-4685-adca-fe27a5664068\" (UID: \"e7a9f323-9e08-4685-adca-fe27a5664068\") " Oct 31 02:07:06.039358 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039337 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.039461 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039374 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-routing-overlay-volume" pod "e7a9f323-9e08-4685-adca-fe27a5664068" (UID: "e7a9f323-9e08-4685-adca-fe27a5664068"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:06.039461 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039383 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-plugins-config-volume" pod "e7a9f323-9e08-4685-adca-fe27a5664068" (UID: "e7a9f323-9e08-4685-adca-fe27a5664068"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:06.039461 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039390 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.039638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039464 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.039833 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.039809 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-kube-api-access-rwm8g\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.040023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.040003 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:06.040121 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.040026 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7a9f323-9e08-4685-adca-fe27a5664068-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:06.041479 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.041460 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-kube-api-access-6mhzb" pod "e7a9f323-9e08-4685-adca-fe27a5664068" (UID: "e7a9f323-9e08-4685-adca-fe27a5664068"). InnerVolumeSpecName "kube-api-access-6mhzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:06.041566 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.041485 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-provider-credentials" pod "e7a9f323-9e08-4685-adca-fe27a5664068" (UID: "e7a9f323-9e08-4685-adca-fe27a5664068"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:06.073682 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.073648 5090 generic.go:356] "Generic (PLEG): container finished" podID="e7a9f323-9e08-4685-adca-fe27a5664068" containerID="2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1" exitCode=0 Oct 31 02:07:06.073843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.073700 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" Oct 31 02:07:06.073843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.073784 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" event={"ID":"e7a9f323-9e08-4685-adca-fe27a5664068","Type":"ContainerDied","Data":"2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1"} Oct 31 02:07:06.073843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.073835 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm" event={"ID":"e7a9f323-9e08-4685-adca-fe27a5664068","Type":"ContainerDied","Data":"d479e4f48c83fb65670d74d6531435319bf8c23af0689ef1be4cf058791efd5a"} Oct 31 02:07:06.074003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.073857 5090 scope.go:122] "RemoveContainer" containerID="2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1" Oct 31 02:07:06.085272 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.085255 5090 scope.go:122] "RemoveContainer" containerID="2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1" Oct 31 02:07:06.085571 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:07:06.085552 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1\": container with ID starting with 2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1 not found: ID does not exist" containerID="2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1" Oct 31 02:07:06.085660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.085580 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1"} err="failed to get container status \"2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1\": rpc error: code = NotFound desc = could not find container \"2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1\": container with ID starting with 2a7636085621f911e186018265812025ba9b1051991f7951da1246534fe739c1 not found: ID does not exist" Oct 31 02:07:06.098869 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.098847 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm"] Oct 31 02:07:06.102501 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.102480 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-877bgmjfm"] Oct 31 02:07:06.141233 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.141174 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-kube-api-access-rwm8g\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.141339 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.141252 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.141427 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.141404 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.141486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.141444 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.141620 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.141606 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mhzb\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-kube-api-access-6mhzb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:06.141660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.141623 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7a9f323-9e08-4685-adca-fe27a5664068-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:06.141953 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.141934 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.142075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.142051 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.144107 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.144089 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.153592 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.153574 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-kube-api-access-rwm8g\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.274866 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.274825 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:06.475235 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.466363 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:06.569634 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:06.569608 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7"] Oct 31 02:07:06.570474 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:07:06.570443 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1a82cb_238f_4f9d_bef8_79220ff49ae7.slice/crio-5c85f36719d821c62ad963f981a518ab0e48fe51d69c5cd3dcf72e934621ea73 WatchSource:0}: Error finding container 5c85f36719d821c62ad963f981a518ab0e48fe51d69c5cd3dcf72e934621ea73: Status 404 returned error can't find the container with id 5c85f36719d821c62ad963f981a518ab0e48fe51d69c5cd3dcf72e934621ea73 Oct 31 02:07:07.095372 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:07.095335 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" event={"ID":"af1a82cb-238f-4f9d-bef8-79220ff49ae7","Type":"ContainerStarted","Data":"b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762"} Oct 31 02:07:07.095804 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:07.095410 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" event={"ID":"af1a82cb-238f-4f9d-bef8-79220ff49ae7","Type":"ContainerStarted","Data":"5c85f36719d821c62ad963f981a518ab0e48fe51d69c5cd3dcf72e934621ea73"} Oct 31 02:07:07.095804 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:07.095448 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:07.121488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:07.121442 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" podStartSLOduration=16.121429941 podStartE2EDuration="16.121429941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:07.119613129 +0000 UTC m=+2827.499858353" watchObservedRunningTime="2026-10-31 02:07:07.121429941 +0000 UTC m=+2827.501675164" Oct 31 02:07:07.212435 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:07.212397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:07.749205 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:07.749171 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a9f323-9e08-4685-adca-fe27a5664068" path="/var/lib/kubelet/pods/e7a9f323-9e08-4685-adca-fe27a5664068/volumes" Oct 31 02:07:07.872459 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:07.872343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:08.568661 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:08.568623 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:09.220429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:09.220396 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:09.883792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:09.875043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:10.574674 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:10.574617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:11.314141 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:11.314104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:12.116352 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:12.116258 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:12.861811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:12.861780 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:13.616385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:13.616347 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:14.325805 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:14.325765 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:15.053454 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:15.051751 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:16.057996 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:16.057965 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:16.206083 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:16.206048 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:16.863731 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:16.863688 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:17.009308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:17.009267 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:17.225792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:17.223299 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:17.906916 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:17.906879 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:18.105843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:18.105807 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:18.588837 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:18.588789 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:19.236120 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:19.236082 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:19.907844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:19.907786 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:20.701428 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:20.701362 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:21.384705 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:21.384656 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:22.058873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:22.058844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:22.808086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:22.808050 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:23.668064 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:23.668030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:24.544510 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:24.544397 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:25.341429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:25.341391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:26.062637 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:26.062602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:26.706002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:26.705894 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:27.554226 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:27.554122 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:28.213027 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:28.212991 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:28.817734 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:28.817684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:29.506076 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:29.505969 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:30.173594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:30.173556 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:30.755442 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:30.755414 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:31.562090 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:31.560964 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:32.256897 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:32.256863 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:33.057752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:33.056544 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:33.411767 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:33.411606 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:33.737882 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:33.737796 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:34.112437 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:34.112406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:34.267283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:34.267250 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:34.919993 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:07:34.919956 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:07:34.977441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:34.977404 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:35.661204 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:35.661171 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:36.353122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:36.353088 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:37.011992 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:37.011958 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:37.666043 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:37.666004 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:37.807773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:37.807703 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:38.045921 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:38.043691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:38.781025 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:38.780988 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:38.923572 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:38.923542 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:39.091794 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:39.089364 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:39.519639 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:39.519609 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:39.680130 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:39.680091 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:40.349942 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:40.349869 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:40.993056 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:40.992991 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:41.646175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:41.646129 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:42.452152 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:42.452114 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:43.136407 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:43.136374 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:43.863066 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:43.863010 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:44.547752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:44.547691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:45.282435 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:45.282392 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:45.994710 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:45.994669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:46.376058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:46.375973 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:46.525259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:46.525227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:46.662950 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:46.662853 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:46.946781 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:46.946687 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7"] Oct 31 02:07:46.953402 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:46.953371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:47.192994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:47.192957 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:47.603416 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:47.603385 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:47.952234 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:47.952153 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:48.752134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:48.752099 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:49.459663 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:49.459627 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:50.266924 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:50.266837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:50.901078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:50.901045 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:51.565290 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:51.565256 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:51.805391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:51.805330 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:51.947629 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:51.947551 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" podUID="af1a82cb-238f-4f9d-bef8-79220ff49ae7" containerName="payload-processing" containerID="cri-o://b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762" gracePeriod=25 Oct 31 02:07:52.235005 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.234979 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:52.269183 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.269155 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6"] Oct 31 02:07:52.270156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.270111 5090 cpu_manager.go:414] "containerMap: removing container" podUID="af1a82cb-238f-4f9d-bef8-79220ff49ae7" containerName="payload-processing" Oct 31 02:07:52.270349 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.270325 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1a82cb-238f-4f9d-bef8-79220ff49ae7" containerName="payload-processing" Oct 31 02:07:52.270556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.270536 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1a82cb-238f-4f9d-bef8-79220ff49ae7" containerName="payload-processing" Oct 31 02:07:52.274491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.274474 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.282023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.281992 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-20538a\"" Oct 31 02:07:52.282294 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.282249 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"agc-cred-c47ea10560ab\"" Oct 31 02:07:52.286246 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.282251 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"agc-cred-70cd9b5e0cb1\"" Oct 31 02:07:52.286246 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.282283 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"openshift-service-ca.crt\"" Oct 31 02:07:52.286246 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.282288 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"routing-overlay\"" Oct 31 02:07:52.286246 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.282369 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"kube-root-ca.crt\"" Oct 31 02:07:52.286246 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.285806 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"payload-processing-external-model-e2e--968a6914-dockercfg-99978\"" Oct 31 02:07:52.286523 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.286304 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6"] Oct 31 02:07:52.344445 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.344111 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:52.361299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.361261 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-kube-api-access-rwm8g\") pod \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " Oct 31 02:07:52.361553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.361517 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-provider-credentials\") pod \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " Oct 31 02:07:52.361745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.361562 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-routing-overlay-volume\") pod \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " Oct 31 02:07:52.361745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.361608 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-plugins-config-volume\") pod \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\" (UID: \"af1a82cb-238f-4f9d-bef8-79220ff49ae7\") " Oct 31 02:07:52.362165 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.362126 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-routing-overlay-volume" pod "af1a82cb-238f-4f9d-bef8-79220ff49ae7" (UID: "af1a82cb-238f-4f9d-bef8-79220ff49ae7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:52.362269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.362223 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-plugins-config-volume" pod "af1a82cb-238f-4f9d-bef8-79220ff49ae7" (UID: "af1a82cb-238f-4f9d-bef8-79220ff49ae7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:07:52.363204 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.363135 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:52.363204 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.363168 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af1a82cb-238f-4f9d-bef8-79220ff49ae7-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:52.364259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.364238 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-provider-credentials" pod "af1a82cb-238f-4f9d-bef8-79220ff49ae7" (UID: "af1a82cb-238f-4f9d-bef8-79220ff49ae7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:52.364753 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.364692 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-kube-api-access-rwm8g" pod "af1a82cb-238f-4f9d-bef8-79220ff49ae7" (UID: "af1a82cb-238f-4f9d-bef8-79220ff49ae7"). InnerVolumeSpecName "kube-api-access-rwm8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:07:52.464430 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.464323 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8nq\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-kube-api-access-fx8nq\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.464598 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.464509 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.464598 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.464548 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.464781 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.464755 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.464907 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.464898 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:52.464954 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.464910 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwm8g\" (UniqueName: \"kubernetes.io/projected/af1a82cb-238f-4f9d-bef8-79220ff49ae7-kube-api-access-rwm8g\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:07:52.514584 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.514554 5090 generic.go:356] "Generic (PLEG): container finished" podID="af1a82cb-238f-4f9d-bef8-79220ff49ae7" containerID="b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762" exitCode=0 Oct 31 02:07:52.514782 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.514589 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" event={"ID":"af1a82cb-238f-4f9d-bef8-79220ff49ae7","Type":"ContainerDied","Data":"b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762"} Oct 31 02:07:52.514782 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.514602 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" Oct 31 02:07:52.514782 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.514625 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7" event={"ID":"af1a82cb-238f-4f9d-bef8-79220ff49ae7","Type":"ContainerDied","Data":"5c85f36719d821c62ad963f981a518ab0e48fe51d69c5cd3dcf72e934621ea73"} Oct 31 02:07:52.514782 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.514645 5090 scope.go:122] "RemoveContainer" containerID="b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762" Oct 31 02:07:52.528865 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.528844 5090 scope.go:122] "RemoveContainer" containerID="b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762" Oct 31 02:07:52.529343 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:07:52.529270 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762\": container with ID starting with b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762 not found: ID does not exist" containerID="b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762" Oct 31 02:07:52.529343 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.529312 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762"} err="failed to get container status \"b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762\": rpc error: code = NotFound desc = could not find container \"b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762\": container with ID starting with b8194f0f3dda9d0b80e582f5c2178db0d5938f763f2634e271eb51f27a7f8762 not found: ID does not exist" Oct 31 02:07:52.541770 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.541745 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7"] Oct 31 02:07:52.545974 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.545953 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-b67fp8vg7"] Oct 31 02:07:52.565794 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.565765 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.566116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.565876 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8nq\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-kube-api-access-fx8nq\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.566116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.566018 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.566116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.566061 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.566657 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.566633 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.566825 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.566808 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.568493 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.568475 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.577314 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.577294 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8nq\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-kube-api-access-fx8nq\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.592917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.592898 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:52.880188 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:52.880162 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6"] Oct 31 02:07:52.881802 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:07:52.881768 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16968ea_159a_4ecf_8370_690179a14cd0.slice/crio-6afd8b09418a99b48542c388d7a4aa28fb022d0617e1b66b0db0d53a53265e8e WatchSource:0}: Error finding container 6afd8b09418a99b48542c388d7a4aa28fb022d0617e1b66b0db0d53a53265e8e: Status 404 returned error can't find the container with id 6afd8b09418a99b48542c388d7a4aa28fb022d0617e1b66b0db0d53a53265e8e Oct 31 02:07:53.123844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:53.123815 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:53.523551 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:53.523516 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" event={"ID":"a16968ea-159a-4ecf-8370-690179a14cd0","Type":"ContainerStarted","Data":"99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970"} Oct 31 02:07:53.523788 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:53.523571 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" event={"ID":"a16968ea-159a-4ecf-8370-690179a14cd0","Type":"ContainerStarted","Data":"6afd8b09418a99b48542c388d7a4aa28fb022d0617e1b66b0db0d53a53265e8e"} Oct 31 02:07:53.523788 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:53.523620 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:07:53.545831 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:53.545787 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" podStartSLOduration=49.545774439 podStartE2EDuration="49.545774439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:07:53.542624497 +0000 UTC m=+2873.922869738" watchObservedRunningTime="2026-10-31 02:07:53.545774439 +0000 UTC m=+2873.926019662" Oct 31 02:07:53.749746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:53.749491 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1a82cb-238f-4f9d-bef8-79220ff49ae7" path="/var/lib/kubelet/pods/af1a82cb-238f-4f9d-bef8-79220ff49ae7/volumes" Oct 31 02:07:53.870739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:53.869247 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:54.029649 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:54.029601 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:54.938123 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:54.938068 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:55.082665 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:55.082627 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:55.407041 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:55.406977 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:55.551254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:55.551224 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:55.699701 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:55.699527 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:55.993421 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:55.993385 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:56.138895 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:56.138860 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:56.278849 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:56.278652 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:56.878741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:56.877026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:57.200604 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:57.200522 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:57.451097 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:57.451012 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:58.156344 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:58.156309 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:58.870194 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:58.869630 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:07:59.564346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:07:59.564314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:00.250818 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:00.250784 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:00.950705 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:00.950666 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:01.730359 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:01.723361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:02.361600 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:02.360987 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:02.543414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:02.543380 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:02.687963 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:02.687863 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:03.176740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:03.176597 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:03.484197 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:03.484086 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:03.641408 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:03.641326 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:04.317743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:04.314845 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:04.546613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:04.546585 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:08:04.713999 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:04.713907 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:04.799454 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:08:04.799415 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:08:04.873052 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:04.872991 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:05.466441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:05.466388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:05.807614 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:05.807575 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:06.009431 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:06.009383 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:06.282616 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:06.282581 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:06.814115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:06.814080 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:07.333782 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:07.333736 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:07.508063 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:07.508030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:07.674546 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:07.674449 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:08.212340 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:08.212313 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:08.429692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:08.429643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:09.168701 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:09.168669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:10.150297 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:10.149780 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:10.829286 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:10.829257 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:11.592681 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:11.592630 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:12.350067 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:12.350032 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:13.054045 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:13.053983 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:13.210307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:13.210266 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:13.919378 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:13.919342 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:14.549344 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:14.549308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:14.953252 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:14.953181 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:15.088792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:15.088762 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:15.336182 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:15.336150 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:15.485702 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:15.485668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:16.045274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:16.045243 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:16.230442 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:16.230392 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:16.381597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:16.381510 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:16.706761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:16.706597 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:16.852336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:16.852292 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:17.285244 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:17.285209 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:17.432090 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:17.432016 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:17.959216 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:17.959115 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:18.587576 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:18.587537 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:18.864930 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:18.859987 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:19.067122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:19.067084 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:19.621297 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:19.620824 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:20.505292 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:20.505252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:21.158574 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:21.158536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:21.681092 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:21.681064 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:21.829024 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:21.828989 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:22.440069 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:22.440021 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:22.590251 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:22.590162 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:22.790593 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:22.790518 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:23.638198 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:23.638073 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:24.322494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:24.322438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:25.090115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:25.090062 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:25.862821 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:25.862768 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:26.510835 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:26.510790 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:27.353804 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:27.353770 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:28.059984 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:28.059901 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:28.614202 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:28.614160 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:28.755269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:28.755236 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:29.699427 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:29.699373 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:30.406413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:30.406373 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:31.257605 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:31.252981 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:32.437740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:32.436789 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:33.560078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:33.560004 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:34.332695 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:34.332657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:34.790517 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:08:34.790473 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:08:34.978745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:34.975189 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:35.556237 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:35.556200 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:36.161210 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:36.161167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:36.829396 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:36.829361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:37.552309 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:37.552269 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:38.265739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:38.265587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:38.856922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:38.856889 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:39.562763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:39.560504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:40.246934 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:40.246687 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:40.894767 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:40.894730 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:41.540189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:41.540148 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:42.168225 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:42.168153 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:42.820118 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:42.820082 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:43.457003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:43.456062 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:44.131115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:44.131080 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:44.931751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:44.931370 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:45.538396 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:45.538340 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:46.152009 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:46.151955 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:46.840523 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:46.840490 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:47.540963 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:47.540928 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:48.257429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:48.257390 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:48.956699 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:48.956661 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:49.560312 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:49.560279 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:50.209878 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:50.203341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:50.850366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:50.850303 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:51.539122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:51.539083 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:51.845771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:51.845664 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:52.168664 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:52.168535 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:52.880969 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:52.880928 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:53.830942 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:53.830907 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:54.539579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:54.539547 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:55.349441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:55.349391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:56.195756 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:56.195726 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:57.307609 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:57.306369 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:58.063277 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:58.063243 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:58.841908 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:58.841875 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:08:59.507664 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:08:59.507612 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:00.173972 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:00.173914 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:00.864751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:00.860229 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:01.712955 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:01.712922 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:02.333501 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:02.333460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:02.992318 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:02.989764 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:03.752369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:03.752274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:04.410127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:04.396939 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:04.790822 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:09:04.790779 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:09:05.099333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:05.099245 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:05.815265 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:05.815227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:06.460928 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:06.460890 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:07.168268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:07.168233 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:07.873546 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:07.869828 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:08.567758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:08.564007 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:09.259493 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:09.259459 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:09.916970 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:09.916932 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:10.668117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:10.668084 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:11.366014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:11.365979 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:12.022259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:12.022221 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:12.648858 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:12.648799 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:12.989980 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:12.989865 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:13.124145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:13.124096 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:13.278926 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:13.278888 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:13.918539 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:13.918508 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:14.556575 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:14.556536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:15.248796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:15.248762 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:15.958738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:15.958341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:16.561677 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:16.561643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:17.261173 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:17.261136 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:17.949809 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:17.949701 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:18.605627 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:18.605589 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:19.351702 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:19.351666 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:19.999283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:19.999230 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:20.692787 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:20.692678 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:21.334534 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:21.334496 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:22.048015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:22.047961 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:22.730978 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:22.730937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:23.365107 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:23.355486 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:24.149336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:24.149248 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:24.820276 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:24.820239 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:25.481385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:25.481346 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:26.265455 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:26.265419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:26.995695 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:26.995627 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:27.670675 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:27.670637 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:28.275303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:28.273072 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:28.989541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:28.989508 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:29.672572 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:29.672535 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:30.362058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:30.354359 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:31.076619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:31.076581 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:31.761692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:31.761658 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:32.485602 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:32.485568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:33.137379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:33.137323 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:33.838377 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:33.838338 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:34.535471 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:34.535428 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:34.790920 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:09:34.790831 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:09:35.177066 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:35.176975 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:35.944729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:35.944650 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:36.641254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:36.641220 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:37.327249 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:37.327216 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:38.148342 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:38.148270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:38.859853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:38.859818 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:39.547198 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:39.547164 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:40.233460 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:40.233420 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:40.853042 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:40.853009 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:41.526469 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:41.526419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:42.175303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:42.175268 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:42.953517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:42.949947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:43.558567 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:43.558518 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:44.182082 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:44.181870 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:44.847401 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:44.847351 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:45.476613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:45.476564 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:46.190549 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:46.190515 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:46.870975 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:46.870934 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:47.547774 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:47.534334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:48.177984 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:48.177892 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:48.940057 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:48.939920 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:49.784127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:49.784089 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:50.166890 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:50.166807 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:50.317486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:50.317442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:50.629002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:50.628965 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:51.218512 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:51.218477 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6"] Oct 31 02:09:51.276278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:51.276241 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:51.426342 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:51.426303 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:51.641278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:51.641239 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:51.910880 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:51.910795 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:52.057444 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:52.057412 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:52.355679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:52.355638 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:52.963076 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:52.963043 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:53.688740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:53.672976 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:54.341363 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:54.341324 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:54.986073 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:54.986032 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:55.815277 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:55.815238 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:56.219972 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.219883 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" podUID="a16968ea-159a-4ecf-8370-690179a14cd0" containerName="payload-processing" containerID="cri-o://99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970" gracePeriod=25 Oct 31 02:09:56.508363 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.508337 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:09:56.555206 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.554987 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:56.559673 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.559625 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld"] Oct 31 02:09:56.560870 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.560846 5090 cpu_manager.go:414] "containerMap: removing container" podUID="a16968ea-159a-4ecf-8370-690179a14cd0" containerName="payload-processing" Oct 31 02:09:56.561060 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.561045 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a16968ea-159a-4ecf-8370-690179a14cd0" containerName="payload-processing" Oct 31 02:09:56.561405 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.561386 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="a16968ea-159a-4ecf-8370-690179a14cd0" containerName="payload-processing" Oct 31 02:09:56.566189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.566166 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.572136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.572114 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"openshift-service-ca.crt\"" Oct 31 02:09:56.573018 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.572997 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"routing-overlay\"" Oct 31 02:09:56.573132 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.573119 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-36d9b2f51461\"" Oct 31 02:09:56.573355 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.573336 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-e2e--c450cd8d-dockercfg-8zv6g\"" Oct 31 02:09:56.573429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.573417 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"kube-root-ca.crt\"" Oct 31 02:09:56.573520 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.573502 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0b6abf\"" Oct 31 02:09:56.586768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.581518 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld"] Oct 31 02:09:56.656367 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.656340 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-provider-credentials\") pod \"a16968ea-159a-4ecf-8370-690179a14cd0\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " Oct 31 02:09:56.656560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.656381 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-routing-overlay-volume\") pod \"a16968ea-159a-4ecf-8370-690179a14cd0\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " Oct 31 02:09:56.656560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.656420 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-kube-api-access-fx8nq\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-kube-api-access-fx8nq\") pod \"a16968ea-159a-4ecf-8370-690179a14cd0\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " Oct 31 02:09:56.656560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.656520 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-plugins-config-volume\") pod \"a16968ea-159a-4ecf-8370-690179a14cd0\" (UID: \"a16968ea-159a-4ecf-8370-690179a14cd0\") " Oct 31 02:09:56.656919 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.656894 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-routing-overlay-volume" pod "a16968ea-159a-4ecf-8370-690179a14cd0" (UID: "a16968ea-159a-4ecf-8370-690179a14cd0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:56.657042 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.657026 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-plugins-config-volume" pod "a16968ea-159a-4ecf-8370-690179a14cd0" (UID: "a16968ea-159a-4ecf-8370-690179a14cd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:09:56.657275 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.657259 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:09:56.657321 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.657278 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a16968ea-159a-4ecf-8370-690179a14cd0-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:09:56.659391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.659366 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-provider-credentials" pod "a16968ea-159a-4ecf-8370-690179a14cd0" (UID: "a16968ea-159a-4ecf-8370-690179a14cd0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:56.659480 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.659397 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-kube-api-access-fx8nq" pod "a16968ea-159a-4ecf-8370-690179a14cd0" (UID: "a16968ea-159a-4ecf-8370-690179a14cd0"). InnerVolumeSpecName "kube-api-access-fx8nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:09:56.758161 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.758130 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.758356 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.758266 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.758426 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.758409 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.758478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.758460 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mt7\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-kube-api-access-69mt7\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.758612 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.758597 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:09:56.758669 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.758614 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx8nq\" (UniqueName: \"kubernetes.io/projected/a16968ea-159a-4ecf-8370-690179a14cd0-kube-api-access-fx8nq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:09:56.765274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.765250 5090 generic.go:356] "Generic (PLEG): container finished" podID="a16968ea-159a-4ecf-8370-690179a14cd0" containerID="99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970" exitCode=0 Oct 31 02:09:56.765426 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.765288 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" Oct 31 02:09:56.765426 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.765322 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" event={"ID":"a16968ea-159a-4ecf-8370-690179a14cd0","Type":"ContainerDied","Data":"99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970"} Oct 31 02:09:56.765426 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.765376 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6" event={"ID":"a16968ea-159a-4ecf-8370-690179a14cd0","Type":"ContainerDied","Data":"6afd8b09418a99b48542c388d7a4aa28fb022d0617e1b66b0db0d53a53265e8e"} Oct 31 02:09:56.765426 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.765410 5090 scope.go:122] "RemoveContainer" containerID="99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970" Oct 31 02:09:56.776812 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.776796 5090 scope.go:122] "RemoveContainer" containerID="99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970" Oct 31 02:09:56.777100 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:09:56.777075 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970\": container with ID starting with 99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970 not found: ID does not exist" containerID="99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970" Oct 31 02:09:56.777167 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.777109 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970"} err="failed to get container status \"99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970\": rpc error: code = NotFound desc = could not find container \"99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970\": container with ID starting with 99923a58b9d6dcc0c15ebe2f1a38321365ba1d4ea4263388e0e85ab3dc2c1970 not found: ID does not exist" Oct 31 02:09:56.791443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.791420 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6"] Oct 31 02:09:56.795313 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.795290 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-59f6gndh6"] Oct 31 02:09:56.859571 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.859531 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mt7\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-kube-api-access-69mt7\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.859689 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.859624 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.859800 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.859701 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.859884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.859856 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.860434 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.860409 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.860515 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.860414 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.862394 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.862377 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.871263 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.871245 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mt7\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-kube-api-access-69mt7\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:56.898122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:56.898096 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:57.173789 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.155400 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:57.210857 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.210828 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld"] Oct 31 02:09:57.212321 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:09:57.212286 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dff408_6802_4435_a521_cbd7d32eb7d4.slice/crio-eb9b89b7cd4af8845ae670143e58453f90f1fc880a380e458be084c75a0efba0 WatchSource:0}: Error finding container eb9b89b7cd4af8845ae670143e58453f90f1fc880a380e458be084c75a0efba0: Status 404 returned error can't find the container with id eb9b89b7cd4af8845ae670143e58453f90f1fc880a380e458be084c75a0efba0 Oct 31 02:09:57.749416 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.749385 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a16968ea-159a-4ecf-8370-690179a14cd0" path="/var/lib/kubelet/pods/a16968ea-159a-4ecf-8370-690179a14cd0/volumes" Oct 31 02:09:57.775189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.775159 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" event={"ID":"f4dff408-6802-4435-a521-cbd7d32eb7d4","Type":"ContainerStarted","Data":"a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa"} Oct 31 02:09:57.775333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.775208 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" event={"ID":"f4dff408-6802-4435-a521-cbd7d32eb7d4","Type":"ContainerStarted","Data":"eb9b89b7cd4af8845ae670143e58453f90f1fc880a380e458be084c75a0efba0"} Oct 31 02:09:57.775333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.775246 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:09:57.794401 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.794357 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" podStartSLOduration=98.794345301 podStartE2EDuration="1m38.794345301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:09:57.793108169 +0000 UTC m=+2998.173353393" watchObservedRunningTime="2026-10-31 02:09:57.794345301 +0000 UTC m=+2998.174590573" Oct 31 02:09:57.837909 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:57.837863 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:58.494748 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:58.491320 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:58.664236 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:58.664204 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:58.811672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:58.811587 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:59.105185 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:59.102843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:09:59.750472 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:09:59.750426 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:00.376216 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:00.376188 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:10:00.381274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:00.381251 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:10:00.433557 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:00.433505 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:00.473658 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:00.473604 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:10:00.479600 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:00.479580 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:10:01.061563 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:01.061528 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:01.782809 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:01.779995 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:02.532064 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:02.532021 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:03.172268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:03.172232 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:03.833942 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:03.833889 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:04.577162 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:04.577074 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:04.791041 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:10:04.790991 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:10:05.221560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:05.221512 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:05.376882 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:05.376843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:05.999532 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:05.999471 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:06.215675 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:06.215617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:06.366799 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:06.365836 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:07.198822 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:07.198785 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:07.816975 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:07.816944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:08.690316 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:08.690277 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:08.786857 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:08.786821 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:10:09.328015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:09.327982 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:09.990742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:09.990456 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:10.603685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:10.603649 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:11.268351 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:11.268317 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:11.958173 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:11.958133 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:12.624390 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:12.624334 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:13.343331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:13.343298 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:14.011446 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:14.011419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:14.679900 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:14.679866 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:15.276335 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:15.276303 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:15.996316 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:15.996281 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:16.614044 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:16.613994 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:17.310989 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:17.310958 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:17.958854 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:17.957581 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:18.570784 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:18.570747 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:19.254027 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:19.253986 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:19.979298 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:19.979259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:20.565652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:20.565612 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:21.348572 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:21.348536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:22.069789 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:22.067887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:22.817195 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:22.817155 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:23.489663 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:23.489627 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:24.255343 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:24.255312 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:25.335314 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:25.335278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:26.124333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:26.124280 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:27.065062 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:27.065015 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:27.717845 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:27.717812 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:28.358784 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:28.358582 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:28.972709 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:28.972677 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:29.623181 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:29.623144 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:30.275984 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:30.275870 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:31.028442 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:31.028407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:31.652799 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:31.652185 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:32.268259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:32.268185 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:32.934233 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:32.934150 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:33.592836 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:33.592803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:34.390738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:34.387684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:34.789225 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:10:34.789185 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:10:35.163187 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:35.163096 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:35.738605 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:35.738544 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:36.362803 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:36.362768 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:37.178485 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:37.178442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:37.834914 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:37.834884 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:38.558454 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:38.558410 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:39.207694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:39.207656 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:39.962861 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:39.962827 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:40.680739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:40.677651 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:41.256469 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:41.256437 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:41.889534 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:41.889504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:42.668682 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:42.668585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:43.311317 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:43.311281 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85"] Oct 31 02:10:43.311813 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:43.311540 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" containerID="cri-o://3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005" gracePeriod=30 Oct 31 02:10:43.372311 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:43.372261 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:44.035231 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:44.035197 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:44.686618 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:44.686586 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:45.375741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:45.375597 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:45.955626 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:45.955591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:46.690075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:46.690030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:47.390086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:47.390050 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:48.039087 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.039055 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:48.598652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.598594 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 02:10:48.635166 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.635130 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7"] Oct 31 02:10:48.636348 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.636322 5090 cpu_manager.go:414] "containerMap: removing container" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" Oct 31 02:10:48.636441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.636350 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" Oct 31 02:10:48.636546 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.636532 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerName="istio-proxy" Oct 31 02:10:48.640787 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.640768 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.650987 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.649656 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-bb5c51-openshift-default-dockercfg-s2hdn\"" Oct 31 02:10:48.660918 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.660888 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7"] Oct 31 02:10:48.753585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.753553 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-podinfo\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.753820 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.753650 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-token\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-token\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.753820 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.753682 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-socket\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.753820 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.753783 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-data\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.754039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.753846 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-credential-socket\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.754039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.753901 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-certs\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.754149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754049 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-envoy\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.754149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754078 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-kube-api-access-cmkvc\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-kube-api-access-cmkvc\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.754149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754080 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-data" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:48.754149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754094 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-socket" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:48.754404 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754151 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d5940c-4db0-471a-a110-c91ff831ae1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d5940c-4db0-471a-a110-c91ff831ae1b-istiod-ca-cert\") pod \"26d5940c-4db0-471a-a110-c91ff831ae1b\" (UID: \"26d5940c-4db0-471a-a110-c91ff831ae1b\") " Oct 31 02:10:48.754404 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754173 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-credential-socket" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:48.754404 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754221 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-certs" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:48.754686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754426 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.754686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754487 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.754686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754552 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d5940c-4db0-471a-a110-c91ff831ae1b-istiod-ca-cert" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:10:48.754686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754591 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjsj8\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-kube-api-access-pjsj8\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.754686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754630 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.755058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754705 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a52edf8-5178-419a-a452-19e019e5587a-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.755058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754820 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a52edf8-5178-419a-a452-19e019e5587a-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.755058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.754900 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.755223 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.755058 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.755223 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.755136 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.755337 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.755272 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.755337 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.755291 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.755337 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.755308 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.755337 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.755323 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.755551 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.755338 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26d5940c-4db0-471a-a110-c91ff831ae1b-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.756527 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.756500 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-token" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:48.756527 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.756509 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-podinfo" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:10:48.756807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.756787 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-kube-api-access-cmkvc" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "kube-api-access-cmkvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:10:48.756807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.756787 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-envoy" pod "26d5940c-4db0-471a-a110-c91ff831ae1b" (UID: "26d5940c-4db0-471a-a110-c91ff831ae1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:10:48.836172 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.836119 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858407 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsj8\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-kube-api-access-pjsj8\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858469 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858548 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a52edf8-5178-419a-a452-19e019e5587a-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858630 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a52edf8-5178-419a-a452-19e019e5587a-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858709 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858782 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858845 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858906 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.858938 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.859024 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.859042 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.859064 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26d5940c-4db0-471a-a110-c91ff831ae1b-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.859084 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkvc\" (UniqueName: \"kubernetes.io/projected/26d5940c-4db0-471a-a110-c91ff831ae1b-kube-api-access-cmkvc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.859567 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.860380 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-data\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.863941 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-credential-socket\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.864553 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a52edf8-5178-419a-a452-19e019e5587a-istiod-ca-cert\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.866840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.865152 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-certs\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.871544 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.871430 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a52edf8-5178-419a-a452-19e019e5587a-istio-podinfo\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.871691 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.871576 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-envoy\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.873482 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.873433 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsj8\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-kube-api-access-pjsj8\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.877856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.877814 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-istio-token\") pod \"e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:48.962758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:48.962685 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:49.255601 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.255568 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7"] Oct 31 02:10:49.256426 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:10:49.256399 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a52edf8_5178_419a_a452_19e019e5587a.slice/crio-d4b7148708c71bd85fa64543cc2867a0ec2bdcf7211055939d73611b5ee87d7a WatchSource:0}: Error finding container d4b7148708c71bd85fa64543cc2867a0ec2bdcf7211055939d73611b5ee87d7a: Status 404 returned error can't find the container with id d4b7148708c71bd85fa64543cc2867a0ec2bdcf7211055939d73611b5ee87d7a Oct 31 02:10:49.259265 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.259228 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:10:49.259365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.259312 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:10:49.259429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.259360 5090 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 31 02:10:49.298004 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.297973 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" event={"ID":"9a52edf8-5178-419a-a452-19e019e5587a","Type":"ContainerStarted","Data":"d4b7148708c71bd85fa64543cc2867a0ec2bdcf7211055939d73611b5ee87d7a"} Oct 31 02:10:49.299431 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.299408 5090 generic.go:356] "Generic (PLEG): container finished" podID="26d5940c-4db0-471a-a110-c91ff831ae1b" containerID="3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005" exitCode=0 Oct 31 02:10:49.299517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.299454 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" Oct 31 02:10:49.299517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.299499 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" event={"ID":"26d5940c-4db0-471a-a110-c91ff831ae1b","Type":"ContainerDied","Data":"3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005"} Oct 31 02:10:49.299588 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.299555 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85" event={"ID":"26d5940c-4db0-471a-a110-c91ff831ae1b","Type":"ContainerDied","Data":"8ae7ac5c73832c2595e01897a9483f66fc89b0be2376a614b8a62b7f9782f8ce"} Oct 31 02:10:49.299619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.299588 5090 scope.go:122] "RemoveContainer" containerID="3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005" Oct 31 02:10:49.311058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.311034 5090 scope.go:122] "RemoveContainer" containerID="3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005" Oct 31 02:10:49.311432 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:10:49.311363 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005\": container with ID starting with 3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005 not found: ID does not exist" containerID="3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005" Oct 31 02:10:49.311432 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.311404 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005"} err="failed to get container status \"3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005\": rpc error: code = NotFound desc = could not find container \"3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005\": container with ID starting with 3b0307c096c45486bdf46083c235e5a422a7fcfd656223d4a410f35ea1f32005 not found: ID does not exist" Oct 31 02:10:49.326318 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.326294 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85"] Oct 31 02:10:49.331268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.331246 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-tpx85"] Oct 31 02:10:49.572848 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.570892 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:49.751731 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:49.751684 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d5940c-4db0-471a-a110-c91ff831ae1b" path="/var/lib/kubelet/pods/26d5940c-4db0-471a-a110-c91ff831ae1b/volumes" Oct 31 02:10:50.239372 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:50.239123 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:50.314980 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:50.314942 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" event={"ID":"9a52edf8-5178-419a-a452-19e019e5587a","Type":"ContainerStarted","Data":"224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633"} Oct 31 02:10:50.341519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:50.341457 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podStartSLOduration=308.341444039 podStartE2EDuration="5m8.341444039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:10:50.337331612 +0000 UTC m=+3050.717576867" watchObservedRunningTime="2026-10-31 02:10:50.341444039 +0000 UTC m=+3050.721689262" Oct 31 02:10:50.925737 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:50.922190 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:50.963581 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:50.963541 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:50.965840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:50.965805 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 02:10:50.966004 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:50.965883 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 02:10:51.811168 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:51.811082 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:51.963620 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:51.963587 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 02:10:51.964116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:51.963666 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 02:10:51.995741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:51.994818 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:52.563791 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:52.563757 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:52.963333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:52.963258 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 02:10:52.963333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:52.963316 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 02:10:53.220624 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:53.220538 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:53.937740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:53.937626 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:53.963525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:53.963493 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 02:10:53.963708 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:53.963551 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 02:10:54.790456 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:54.790420 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:54.963164 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:54.963135 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 31 02:10:54.963334 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:54.963186 5090 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 31 02:10:55.467757 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:55.467410 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:55.969133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:55.969103 5090 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:55.969510 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:55.969375 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:55.970170 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:55.970147 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:10:56.277778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:56.277740 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:56.958949 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:56.958764 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:57.543910 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:57.543873 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:58.329743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:58.329073 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:10:59.196626 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:10:59.196590 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:00.030741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:00.029746 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:00.982899 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:00.982865 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:02.770684 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:02.770641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:02.983739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:02.983657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:03.644245 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:03.644210 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:04.320851 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:04.320802 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:04.793681 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:11:04.793648 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:11:04.990093 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:04.990056 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:05.741375 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:05.741270 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:06.360553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:06.360516 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:07.091741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:07.090374 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:07.777028 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:07.776974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:08.551496 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:08.551445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:09.305746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:09.305694 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:09.957807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:09.957764 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:10.691867 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:10.691829 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:11.284004 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:11.283968 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:11.946889 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:11.946852 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:12.600376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:12.600308 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:13.231434 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:13.231393 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:13.856431 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:13.856369 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:14.623565 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:14.623536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:15.236947 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:15.236913 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:15.869742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:15.861576 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:16.553816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:16.553760 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:17.263763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:17.260325 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:17.885745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:17.883792 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:18.564592 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:18.564546 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:19.296500 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:19.296458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:20.114682 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:20.114639 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:20.808740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:20.802911 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:21.540623 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:21.540588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:22.289164 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:22.289121 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:23.021754 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:23.020642 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:23.637594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:23.637557 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:24.329268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:24.329230 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:24.977232 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:24.977198 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:25.643219 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:25.643187 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:26.351439 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:26.351406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:26.984662 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:26.984583 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:27.664514 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:27.664454 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:28.357243 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:28.357210 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:28.958743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:28.945954 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:29.624478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:29.623884 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:30.227680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:30.227643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:30.976184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:30.976149 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:31.647368 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:31.646772 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:32.484486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:32.484438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:33.084167 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:33.084025 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:33.772643 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:33.764509 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:34.350770 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:34.346833 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:34.792769 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:11:34.792734 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:11:34.958505 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:34.958473 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:35.565932 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:35.565893 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:36.334857 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:36.334813 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:37.004424 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:37.004383 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:37.729958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:37.729924 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:38.423089 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:38.423040 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:39.060864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:39.060826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:39.831338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:39.831304 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:40.145047 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:40.144917 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:40.308738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:40.304383 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:41.220940 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:41.220904 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:41.978768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:41.977550 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:42.739587 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:42.739554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:43.424586 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:43.424557 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:44.080741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:44.080591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:44.740631 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:44.740320 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:45.454841 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:45.446562 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:46.293937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:46.293905 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:46.844905 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:46.844859 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:47.061116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:47.060422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:47.212338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:47.212251 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:48.086933 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:48.086887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:48.980746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:48.980406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:49.772573 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:49.772446 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:50.355611 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:50.355561 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:51.078496 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:51.078458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:51.752405 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:51.752371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:52.043798 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:52.043687 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:52.393558 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:52.393458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:53.249485 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:53.249451 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:53.861332 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:53.860980 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:54.678706 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:54.678638 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:55.455641 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:55.455611 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:56.188999 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:56.188956 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:56.811585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:56.811546 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:57.462814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:57.462760 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:58.139542 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:58.139486 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:58.973473 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:58.973440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:59.214799 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:59.214762 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:59.610050 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:59.609969 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:11:59.874923 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:11:59.874837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:00.676192 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:00.676160 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:01.341274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:01.341194 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:02.041840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:02.041806 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:02.663906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:02.663868 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:03.338405 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:03.338370 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:03.973884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:03.973843 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:04.582736 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:04.581129 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:04.790802 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:12:04.790767 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:12:05.256893 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:05.256158 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:05.974251 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:05.974218 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:06.673053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:06.673011 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:07.386749 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:07.385179 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:08.142823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:08.140557 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:08.777703 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:08.777669 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:09.549788 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:09.549754 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:10.166190 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:10.166154 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:10.830558 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:10.830503 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:11.444642 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:11.444515 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:12.000041 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:11.999999 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw"] Oct 31 02:12:12.000286 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.000262 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" podUID="6967a11e-c22c-4e94-9782-81c9cd8f3cec" containerName="maas-api" containerID="cri-o://8802078ea040e4d8c592cd6c35eee6633eb5ac21aec930ef929e1c52f1b8cb0c" gracePeriod=30 Oct 31 02:12:12.087094 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.087065 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:12.343022 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.326654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:12.343022 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.328298 5090 generic.go:356] "Generic (PLEG): container finished" podID="6967a11e-c22c-4e94-9782-81c9cd8f3cec" containerID="8802078ea040e4d8c592cd6c35eee6633eb5ac21aec930ef929e1c52f1b8cb0c" exitCode=0 Oct 31 02:12:12.343022 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.328491 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" event={"ID":"6967a11e-c22c-4e94-9782-81c9cd8f3cec","Type":"ContainerDied","Data":"8802078ea040e4d8c592cd6c35eee6633eb5ac21aec930ef929e1c52f1b8cb0c"} Oct 31 02:12:12.552409 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.552381 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:12:12.573040 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.572981 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:12.584350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.579727 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6967a11e-c22c-4e94-9782-81c9cd8f3cec-kube-api-access-llxk8\" (UniqueName: \"kubernetes.io/projected/6967a11e-c22c-4e94-9782-81c9cd8f3cec-kube-api-access-llxk8\") pod \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " Oct 31 02:12:12.584350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.579845 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-maas-api-tls\") pod \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " Oct 31 02:12:12.584350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.579881 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6967a11e-c22c-4e94-9782-81c9cd8f3cec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6967a11e-c22c-4e94-9782-81c9cd8f3cec-openshift-service-ca\") pod \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " Oct 31 02:12:12.584350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.579903 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-metrics-certs\") pod \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\" (UID: \"6967a11e-c22c-4e94-9782-81c9cd8f3cec\") " Oct 31 02:12:12.585773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.585514 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6967a11e-c22c-4e94-9782-81c9cd8f3cec-openshift-service-ca" pod "6967a11e-c22c-4e94-9782-81c9cd8f3cec" (UID: "6967a11e-c22c-4e94-9782-81c9cd8f3cec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:12:12.587184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.587141 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw"] Oct 31 02:12:12.588158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.588136 5090 cpu_manager.go:414] "containerMap: removing container" podUID="6967a11e-c22c-4e94-9782-81c9cd8f3cec" containerName="maas-api" Oct 31 02:12:12.588158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.588160 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6967a11e-c22c-4e94-9782-81c9cd8f3cec" containerName="maas-api" Oct 31 02:12:12.588315 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.588302 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="6967a11e-c22c-4e94-9782-81c9cd8f3cec" containerName="maas-api" Oct 31 02:12:12.589513 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.589472 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-maas-api-tls" pod "6967a11e-c22c-4e94-9782-81c9cd8f3cec" (UID: "6967a11e-c22c-4e94-9782-81c9cd8f3cec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:12:12.613101 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.608161 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79"] Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.613531 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6967a11e-c22c-4e94-9782-81c9cd8f3cec-kube-api-access-llxk8" pod "6967a11e-c22c-4e94-9782-81c9cd8f3cec" (UID: "6967a11e-c22c-4e94-9782-81c9cd8f3cec"). InnerVolumeSpecName "kube-api-access-llxk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.613790 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.619482 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.620464 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qsc4d\"" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.620530 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-415db4c48eaf\"" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.620873 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7475ba9a0297\"" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.620988 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f3e2ace89eb9\"" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.621080 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.621136 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 02:12:12.621937 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.621205 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 02:12:12.631673 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.631641 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-metrics-certs" pod "6967a11e-c22c-4e94-9782-81c9cd8f3cec" (UID: "6967a11e-c22c-4e94-9782-81c9cd8f3cec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:12:12.632070 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.632039 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw"] Oct 31 02:12:12.632188 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.632097 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79"] Oct 31 02:12:12.632252 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.632197 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.681407 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681379 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-plugins-config-volume\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.681608 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681499 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjpx\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-kube-api-access-sqjpx\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.681608 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681522 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.681608 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681545 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-provider-credentials\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.681819 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681704 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.681819 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681758 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vtn2\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-kube-api-access-5vtn2\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.681917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681838 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.681917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681895 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-routing-overlay-volume\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.682002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681981 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:12.682002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.681992 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6967a11e-c22c-4e94-9782-81c9cd8f3cec-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:12.682002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.682000 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6967a11e-c22c-4e94-9782-81c9cd8f3cec-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:12.682118 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.682012 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llxk8\" (UniqueName: \"kubernetes.io/projected/6967a11e-c22c-4e94-9782-81c9cd8f3cec-kube-api-access-llxk8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:12.730774 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.728970 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0b6abf\"/\"agc-cred-dd41e9644f67\"" Oct 31 02:12:12.774850 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.774776 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:12.783133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.783108 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjpx\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-kube-api-access-sqjpx\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.783283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.783140 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.783283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.783163 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-provider-credentials\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.783283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.783228 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.783454 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.783281 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vtn2\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-kube-api-access-5vtn2\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.783866 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.783840 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.783961 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.783944 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-routing-overlay-volume\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.784058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.784039 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-plugins-config-volume\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.784168 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.784138 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.784392 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.784368 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.784776 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.784747 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-routing-overlay-volume\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.784883 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.784829 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-plugins-config-volume\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.786259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.786232 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-provider-credentials\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.786500 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.786484 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.804972 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.804947 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjpx\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-kube-api-access-sqjpx\") pod \"payload-processing-external-model-79f59b7c8-2qsvw\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.805061 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.805028 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vtn2\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-kube-api-access-5vtn2\") pod \"payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:12.954696 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.954608 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:12.961254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:12.961226 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:13.119852 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.119820 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:13.268657 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.268544 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw"] Oct 31 02:12:13.270520 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:12:13.270493 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529cc133_aef7_4fa5_872b_c6e1713742be.slice/crio-1520b9b779923cb1fc628101d49c10882a765c6eff8672acc876afd9d454557d WatchSource:0}: Error finding container 1520b9b779923cb1fc628101d49c10882a765c6eff8672acc876afd9d454557d: Status 404 returned error can't find the container with id 1520b9b779923cb1fc628101d49c10882a765c6eff8672acc876afd9d454557d Oct 31 02:12:13.271264 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:12:13.271242 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddcaaed_cfe1_48d6_93be_ae92c74bf3b3.slice/crio-4e3c17ea7cee3b80c78b700e93c0d30611d7568479c7f7094a07550a15280d76 WatchSource:0}: Error finding container 4e3c17ea7cee3b80c78b700e93c0d30611d7568479c7f7094a07550a15280d76: Status 404 returned error can't find the container with id 4e3c17ea7cee3b80c78b700e93c0d30611d7568479c7f7094a07550a15280d76 Oct 31 02:12:13.273881 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.273683 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79"] Oct 31 02:12:13.336276 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.336245 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" event={"ID":"529cc133-aef7-4fa5-872b-c6e1713742be","Type":"ContainerStarted","Data":"1520b9b779923cb1fc628101d49c10882a765c6eff8672acc876afd9d454557d"} Oct 31 02:12:13.337641 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.337610 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" event={"ID":"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3","Type":"ContainerStarted","Data":"4e3c17ea7cee3b80c78b700e93c0d30611d7568479c7f7094a07550a15280d76"} Oct 31 02:12:13.339246 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.339227 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" event={"ID":"6967a11e-c22c-4e94-9782-81c9cd8f3cec","Type":"ContainerDied","Data":"34c70a56bf518dd77e21f30030b54b6fe81f328abbffddbcd0faf58cbd8c61a6"} Oct 31 02:12:13.339346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.339272 5090 scope.go:122] "RemoveContainer" containerID="8802078ea040e4d8c592cd6c35eee6633eb5ac21aec930ef929e1c52f1b8cb0c" Oct 31 02:12:13.339346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.339297 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw" Oct 31 02:12:13.387263 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.387241 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw"] Oct 31 02:12:13.393327 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.393304 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2186c3-977768b65-6gdfw"] Oct 31 02:12:13.463740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.463588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:13.749462 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:13.749428 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6967a11e-c22c-4e94-9782-81c9cd8f3cec" path="/var/lib/kubelet/pods/6967a11e-c22c-4e94-9782-81c9cd8f3cec/volumes" Oct 31 02:12:14.271614 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.271520 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:14.349324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.349291 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" event={"ID":"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3","Type":"ContainerStarted","Data":"02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5"} Oct 31 02:12:14.349547 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.349387 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:14.352413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.352385 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" event={"ID":"529cc133-aef7-4fa5-872b-c6e1713742be","Type":"ContainerStarted","Data":"8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74"} Oct 31 02:12:14.352551 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.352521 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:14.372588 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.372544 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" podStartSLOduration=28.372530897 podStartE2EDuration="28.372530897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:14.367932326 +0000 UTC m=+3134.748177548" watchObservedRunningTime="2026-10-31 02:12:14.372530897 +0000 UTC m=+3134.752776154" Oct 31 02:12:14.390268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.390217 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" podStartSLOduration=33.390205427 podStartE2EDuration="33.390205427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:12:14.386922996 +0000 UTC m=+3134.767168239" watchObservedRunningTime="2026-10-31 02:12:14.390205427 +0000 UTC m=+3134.770450649" Oct 31 02:12:14.952594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:14.952352 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:15.588925 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:15.588887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:16.298748 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:16.295495 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:16.989369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:16.988676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:17.753000 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:17.752953 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:18.447748 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:18.441265 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:19.163931 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:19.163876 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:19.555740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:19.551862 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:19.723971 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:19.723923 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:19.884652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:19.883108 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:20.296556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:20.296521 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:21.533934 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:21.533876 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:22.548350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:22.548246 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:23.184588 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:23.184550 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:23.910753 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:23.909251 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:24.689571 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:24.689534 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:25.366273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:25.366240 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:12:25.367485 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:25.367459 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:12:25.402283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:25.402252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:25.427953 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:25.427919 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld"] Oct 31 02:12:26.299606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:26.298349 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:27.017238 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:27.017205 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:27.758235 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:27.758199 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:28.459666 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:28.459602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:29.138859 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:29.138826 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:29.752983 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:29.752950 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:30.428639 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.428602 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" podUID="f4dff408-6802-4435-a521-cbd7d32eb7d4" containerName="payload-processing" containerID="cri-o://a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa" gracePeriod=25 Oct 31 02:12:30.487412 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.479460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:30.750584 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.750546 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:12:30.784213 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.784185 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:30.788075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.788048 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-routing-overlay-volume\") pod \"f4dff408-6802-4435-a521-cbd7d32eb7d4\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " Oct 31 02:12:30.788236 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.788084 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-provider-credentials\") pod \"f4dff408-6802-4435-a521-cbd7d32eb7d4\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " Oct 31 02:12:30.788236 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.788136 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-kube-api-access-69mt7\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-kube-api-access-69mt7\") pod \"f4dff408-6802-4435-a521-cbd7d32eb7d4\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " Oct 31 02:12:30.788375 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.788291 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-plugins-config-volume\") pod \"f4dff408-6802-4435-a521-cbd7d32eb7d4\" (UID: \"f4dff408-6802-4435-a521-cbd7d32eb7d4\") " Oct 31 02:12:30.789283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.789241 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-plugins-config-volume" pod "f4dff408-6802-4435-a521-cbd7d32eb7d4" (UID: "f4dff408-6802-4435-a521-cbd7d32eb7d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:12:30.789383 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.789350 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-routing-overlay-volume" pod "f4dff408-6802-4435-a521-cbd7d32eb7d4" (UID: "f4dff408-6802-4435-a521-cbd7d32eb7d4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:12:30.791516 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.791490 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-provider-credentials" pod "f4dff408-6802-4435-a521-cbd7d32eb7d4" (UID: "f4dff408-6802-4435-a521-cbd7d32eb7d4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:30.791627 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.791604 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-kube-api-access-69mt7" pod "f4dff408-6802-4435-a521-cbd7d32eb7d4" (UID: "f4dff408-6802-4435-a521-cbd7d32eb7d4"). InnerVolumeSpecName "kube-api-access-69mt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:12:30.890346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.890319 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:30.890346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.890346 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:30.890553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.890357 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69mt7\" (UniqueName: \"kubernetes.io/projected/f4dff408-6802-4435-a521-cbd7d32eb7d4-kube-api-access-69mt7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:30.890553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.890366 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dff408-6802-4435-a521-cbd7d32eb7d4-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:12:30.922535 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:30.922505 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:31.306741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.303125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:31.543263 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.543219 5090 generic.go:356] "Generic (PLEG): container finished" podID="f4dff408-6802-4435-a521-cbd7d32eb7d4" containerID="a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa" exitCode=0 Oct 31 02:12:31.543458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.543281 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" Oct 31 02:12:31.543458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.543285 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" event={"ID":"f4dff408-6802-4435-a521-cbd7d32eb7d4","Type":"ContainerDied","Data":"a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa"} Oct 31 02:12:31.543458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.543353 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld" event={"ID":"f4dff408-6802-4435-a521-cbd7d32eb7d4","Type":"ContainerDied","Data":"eb9b89b7cd4af8845ae670143e58453f90f1fc880a380e458be084c75a0efba0"} Oct 31 02:12:31.543458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.543383 5090 scope.go:122] "RemoveContainer" containerID="a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa" Oct 31 02:12:31.555302 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.555283 5090 scope.go:122] "RemoveContainer" containerID="a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa" Oct 31 02:12:31.555550 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:12:31.555523 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa\": container with ID starting with a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa not found: ID does not exist" containerID="a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa" Oct 31 02:12:31.555610 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.555554 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa"} err="failed to get container status \"a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa\": rpc error: code = NotFound desc = could not find container \"a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa\": container with ID starting with a4dbc23f50e5b6faa45c2f5bcd92f3c8ec7cb56d4f17cce066530b424e24b6fa not found: ID does not exist" Oct 31 02:12:31.568568 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.568494 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld"] Oct 31 02:12:31.572402 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.572378 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5bd6q6dld"] Oct 31 02:12:31.748731 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.748677 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dff408-6802-4435-a521-cbd7d32eb7d4" path="/var/lib/kubelet/pods/f4dff408-6802-4435-a521-cbd7d32eb7d4/volumes" Oct 31 02:12:32.001742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:31.994371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:32.728758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:32.721561 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:33.484752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:33.484343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:34.119563 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:34.119531 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:34.791325 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:12:34.791287 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:12:34.862681 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:34.860834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:35.586738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:35.580460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:36.262738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:36.260558 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:36.889440 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:36.889392 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:37.556745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:37.556691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:38.264442 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:38.264406 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:38.960133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:38.960093 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:39.645318 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:39.645245 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:40.252974 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:40.252883 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:40.964906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:40.963860 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:41.469503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:41.469468 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:41.613331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:41.613295 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:41.768001 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:41.767961 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:41.917746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:41.911091 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:42.188033 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:42.187950 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:42.773740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:42.768498 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:43.444559 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:43.444524 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:44.174579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:44.174526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:44.866079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:44.866034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:45.560839 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:45.560804 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:46.166741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:46.166320 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:46.788356 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:46.788321 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:47.495743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:47.489830 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:48.134734 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:48.134671 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:48.837859 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:48.837827 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:48.987008 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:48.986970 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:49.233271 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:49.233196 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:49.861903 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:49.859823 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:50.547445 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:50.547411 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:51.246058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:51.246003 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:51.924287 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:51.924115 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:52.135354 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:52.135320 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:52.695741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:52.687705 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:53.401525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:53.401331 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:54.056366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:54.056333 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:54.732597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:54.732564 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:55.368743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:55.364263 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:56.196278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:56.196244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:56.797843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:56.796017 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:57.570868 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:57.570833 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:58.148876 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:58.148830 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:58.818346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:58.818312 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:59.437386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:59.437356 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:12:59.981277 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:12:59.981244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:00.124492 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:00.124460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:00.279741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:00.278851 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:00.766458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:00.766425 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:00.905053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:00.905017 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:01.048300 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:01.048180 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:01.752763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:01.752658 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:02.175105 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.175023 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v"] Oct 31 02:13:02.175798 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.175764 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" podUID="7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" containerName="maas-api" containerID="cri-o://c8d2a3967a903babdbe5a15b4e7d8f6d2b0b4bf9fc6769c4d8d1449075b503d6" gracePeriod=30 Oct 31 02:13:02.254793 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.254758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:02.433459 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.433304 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:02.714110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.713997 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:02.829692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.829659 5090 generic.go:356] "Generic (PLEG): container finished" podID="7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" containerID="c8d2a3967a903babdbe5a15b4e7d8f6d2b0b4bf9fc6769c4d8d1449075b503d6" exitCode=0 Oct 31 02:13:02.830154 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.830130 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" event={"ID":"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b","Type":"ContainerDied","Data":"c8d2a3967a903babdbe5a15b4e7d8f6d2b0b4bf9fc6769c4d8d1449075b503d6"} Oct 31 02:13:02.873102 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.873075 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 02:13:02.943818 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.943777 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:02.997266 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.997125 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-kube-api-access-5j5kb\" (UniqueName: \"kubernetes.io/projected/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-kube-api-access-5j5kb\") pod \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " Oct 31 02:13:02.997494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.997274 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-maas-api-tls\") pod \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " Oct 31 02:13:02.997494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.997307 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-metrics-certs\") pod \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " Oct 31 02:13:02.997494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.997439 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-openshift-service-ca\") pod \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\" (UID: \"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b\") " Oct 31 02:13:02.998150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.998117 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-openshift-service-ca" pod "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" (UID: "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:02.999977 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:02.999954 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-kube-api-access-5j5kb" pod "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" (UID: "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b"). InnerVolumeSpecName "kube-api-access-5j5kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:03.000178 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.000159 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-maas-api-tls" pod "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" (UID: "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:03.000430 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.000415 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-metrics-certs" pod "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" (UID: "7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:03.099375 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.099335 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j5kb\" (UniqueName: \"kubernetes.io/projected/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-kube-api-access-5j5kb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:03.099375 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.099375 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:03.099375 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.099385 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:03.099734 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.099394 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:03.287587 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.287511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:03.738792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.738615 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:03.838387 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.838358 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" Oct 31 02:13:03.838558 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.838389 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v" event={"ID":"7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b","Type":"ContainerDied","Data":"cfad7c2e5ae18b8dfe79efb05d4a43f6b799b9064d1420a9f90a05f97efa5b2c"} Oct 31 02:13:03.838558 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.838458 5090 scope.go:122] "RemoveContainer" containerID="c8d2a3967a903babdbe5a15b4e7d8f6d2b0b4bf9fc6769c4d8d1449075b503d6" Oct 31 02:13:03.863983 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.863954 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v"] Oct 31 02:13:03.870138 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:03.870115 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-46a67e-c6569b95c-vs44v"] Oct 31 02:13:04.341763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:04.341733 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:04.572600 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:04.572570 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:04.725681 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:04.725588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:04.938144 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:04.938108 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:13:05.747785 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:05.747754 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" path="/var/lib/kubelet/pods/7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b/volumes" Oct 31 02:13:06.025642 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:06.022547 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:06.780860 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:06.779442 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:07.377756 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:07.363375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:08.187742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:08.184937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:08.904877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:08.904839 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:09.643331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:09.643274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:09.885592 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:09.885532 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:10.036751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:10.036623 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:10.161806 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:10.161773 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7"] Oct 31 02:13:10.162065 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:10.162019 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" containerID="cri-o://ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507" gracePeriod=30 Oct 31 02:13:10.224670 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:10.224639 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:10.577666 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:10.577631 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:11.533974 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:11.533909 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:12.261373 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:12.261333 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:12.508554 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:12.508514 5090 patch_prober.go:29] interesting pod/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:13:12.508780 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:12.508579 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:13:12.806738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:12.799512 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:12.942332 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:12.942283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:13.427744 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:13.424288 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:14.241477 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:14.241423 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:14.879249 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:14.879211 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:15.469743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.469697 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 02:13:15.534409 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534375 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-podinfo\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534429 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-token\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-token\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534465 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-kube-api-access-7xpv6\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-kube-api-access-7xpv6\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534489 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-credential-socket\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534545 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-data\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534628 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-socket\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534665 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istiod-ca-cert\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534700 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-certs\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.534863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.534842 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-envoy\") pod \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\" (UID: \"e20804cd-2d74-4ca2-b037-d8ff00e003d4\") " Oct 31 02:13:15.535274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535164 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-socket" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:15.535274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535177 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-credential-socket" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:15.535274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535244 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istiod-ca-cert" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:15.535541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535412 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-data" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:15.535842 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535731 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.535906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535893 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.535949 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535913 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.535949 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.535928 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.536186 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.536164 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-certs" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:15.537512 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.537489 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-kube-api-access-7xpv6" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "kube-api-access-7xpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:15.537655 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.537635 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-podinfo" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:15.537968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.537946 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-envoy" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:15.538308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.538289 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-token" pod "e20804cd-2d74-4ca2-b037-d8ff00e003d4" (UID: "e20804cd-2d74-4ca2-b037-d8ff00e003d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:15.637982 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.637951 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.637982 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.637985 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.638961 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.638000 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpv6\" (UniqueName: \"kubernetes.io/projected/e20804cd-2d74-4ca2-b037-d8ff00e003d4-kube-api-access-7xpv6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.638961 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.638017 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.638961 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.638030 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e20804cd-2d74-4ca2-b037-d8ff00e003d4-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:15.763080 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.757391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:15.995681 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.995645 5090 generic.go:356] "Generic (PLEG): container finished" podID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerID="ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507" exitCode=0 Oct 31 02:13:15.995864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.995694 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" Oct 31 02:13:15.995864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.995762 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" event={"ID":"e20804cd-2d74-4ca2-b037-d8ff00e003d4","Type":"ContainerDied","Data":"ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507"} Oct 31 02:13:15.995864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.995819 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7" event={"ID":"e20804cd-2d74-4ca2-b037-d8ff00e003d4","Type":"ContainerDied","Data":"1fa8708d249c213d0fefcc99ae832845d1524ca0af75e9dbc9384fde3ea0b553"} Oct 31 02:13:15.995864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:15.995847 5090 scope.go:122] "RemoveContainer" containerID="ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507" Oct 31 02:13:16.006824 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:16.006803 5090 scope.go:122] "RemoveContainer" containerID="ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507" Oct 31 02:13:16.007091 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:16.007070 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507\": container with ID starting with ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507 not found: ID does not exist" containerID="ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507" Oct 31 02:13:16.007162 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:16.007102 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507"} err="failed to get container status \"ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507\": rpc error: code = NotFound desc = could not find container \"ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507\": container with ID starting with ca01785812cabab4b24572ea4778b38bb88167d0386c73b725a05c8ba8dfd507 not found: ID does not exist" Oct 31 02:13:16.014577 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:16.014523 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7"] Oct 31 02:13:16.021295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:16.021272 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-46a67e-openshift-default-bdcdc75df-nxmn7"] Oct 31 02:13:16.524020 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:16.523978 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:17.255051 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:17.254976 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:17.749212 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:17.749184 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" path="/var/lib/kubelet/pods/e20804cd-2d74-4ca2-b037-d8ff00e003d4/volumes" Oct 31 02:13:17.925291 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:17.925259 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:18.559276 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:18.559242 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:19.265750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:19.264959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:19.984253 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:19.984179 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:20.586573 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:20.586533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:20.829729 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:20.829679 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9"] Oct 31 02:13:20.844973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:20.844894 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv"] Oct 31 02:13:20.946934 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:20.946897 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:21.089983 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:21.089944 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:21.382063 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:21.382011 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:22.091831 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:22.091798 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:22.675741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:22.675671 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:25.081836 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:25.081788 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:25.238491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:25.238441 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:25.732377 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:25.732319 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:25.838491 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:25.838441 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" podUID="7bd7c26e-106b-4d86-9885-9026a336feca" containerName="payload-processing" containerID="cri-o://f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0" gracePeriod=25 Oct 31 02:13:25.845803 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:25.845693 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" podUID="4333d31c-1c31-4394-a273-3fabbc1751ca" containerName="payload-pre-processing" containerID="cri-o://7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9" gracePeriod=25 Oct 31 02:13:26.463257 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:26.463223 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:27.177915 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:27.177880 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:27.796244 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:27.796154 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:28.461016 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:28.460980 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:29.180581 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:29.179901 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:29.862917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:29.862882 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:30.356369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:30.356336 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz"] Oct 31 02:13:30.356901 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:30.356568 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" podUID="3aff8334-9229-4a72-89b7-5128f0d59380" containerName="maas-api" containerID="cri-o://a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db" gracePeriod=30 Oct 31 02:13:30.435871 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:30.435837 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:30.635799 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:30.629604 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:30.799742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:30.784593 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:30.962856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:30.962054 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:31.012785 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.012734 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 02:13:31.081599 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.081568 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aff8334-9229-4a72-89b7-5128f0d59380-kube-api-access-659j5\" (UniqueName: \"kubernetes.io/projected/3aff8334-9229-4a72-89b7-5128f0d59380-kube-api-access-659j5\") pod \"3aff8334-9229-4a72-89b7-5128f0d59380\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " Oct 31 02:13:31.081864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.081613 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aff8334-9229-4a72-89b7-5128f0d59380-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff8334-9229-4a72-89b7-5128f0d59380-openshift-service-ca\") pod \"3aff8334-9229-4a72-89b7-5128f0d59380\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " Oct 31 02:13:31.081864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.081661 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-maas-api-tls\") pod \"3aff8334-9229-4a72-89b7-5128f0d59380\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " Oct 31 02:13:31.081864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.081802 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-metrics-certs\") pod \"3aff8334-9229-4a72-89b7-5128f0d59380\" (UID: \"3aff8334-9229-4a72-89b7-5128f0d59380\") " Oct 31 02:13:31.082087 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.082068 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aff8334-9229-4a72-89b7-5128f0d59380-openshift-service-ca" pod "3aff8334-9229-4a72-89b7-5128f0d59380" (UID: "3aff8334-9229-4a72-89b7-5128f0d59380"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:31.082873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.082843 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff8334-9229-4a72-89b7-5128f0d59380-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:31.084581 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.084551 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-maas-api-tls" pod "3aff8334-9229-4a72-89b7-5128f0d59380" (UID: "3aff8334-9229-4a72-89b7-5128f0d59380"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:31.084765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.084574 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aff8334-9229-4a72-89b7-5128f0d59380-kube-api-access-659j5" pod "3aff8334-9229-4a72-89b7-5128f0d59380" (UID: "3aff8334-9229-4a72-89b7-5128f0d59380"). InnerVolumeSpecName "kube-api-access-659j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:31.085537 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.085506 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-metrics-certs" pod "3aff8334-9229-4a72-89b7-5128f0d59380" (UID: "3aff8334-9229-4a72-89b7-5128f0d59380"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:31.158256 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.158219 5090 generic.go:356] "Generic (PLEG): container finished" podID="3aff8334-9229-4a72-89b7-5128f0d59380" containerID="a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db" exitCode=0 Oct 31 02:13:31.158433 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.158261 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" Oct 31 02:13:31.158433 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.158301 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" event={"ID":"3aff8334-9229-4a72-89b7-5128f0d59380","Type":"ContainerDied","Data":"a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db"} Oct 31 02:13:31.158433 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.158358 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz" event={"ID":"3aff8334-9229-4a72-89b7-5128f0d59380","Type":"ContainerDied","Data":"a64bb80412e40cde7c37ebb8f33e074410bb4063c6571610d0da3edb78d949cf"} Oct 31 02:13:31.158433 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.158380 5090 scope.go:122] "RemoveContainer" containerID="a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db" Oct 31 02:13:31.181270 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.181232 5090 scope.go:122] "RemoveContainer" containerID="a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db" Oct 31 02:13:31.182012 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:31.181948 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db\": container with ID starting with a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db not found: ID does not exist" containerID="a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db" Oct 31 02:13:31.182155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.182030 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db"} err="failed to get container status \"a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db\": rpc error: code = NotFound desc = could not find container \"a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db\": container with ID starting with a34253ebca477ba9d712fcc22c8798b8c15538635370de105093c12b104a41db not found: ID does not exist" Oct 31 02:13:31.184894 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.184866 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-659j5\" (UniqueName: \"kubernetes.io/projected/3aff8334-9229-4a72-89b7-5128f0d59380-kube-api-access-659j5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:31.184894 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.184894 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:31.185087 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.184911 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff8334-9229-4a72-89b7-5128f0d59380-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:31.194962 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.194936 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz"] Oct 31 02:13:31.198255 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.198230 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-6c54a1-6d65558dd9-l46hz"] Oct 31 02:13:31.211827 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.211802 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:31.792394 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.792353 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:31.810336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:31.810297 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aff8334-9229-4a72-89b7-5128f0d59380" path="/var/lib/kubelet/pods/3aff8334-9229-4a72-89b7-5128f0d59380/volumes" Oct 31 02:13:32.397746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:32.395526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:32.957949 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:32.957912 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" podUID="7bd7c26e-106b-4d86-9885-9026a336feca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:32.958431 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:32.958147 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" podUID="4333d31c-1c31-4394-a273-3fabbc1751ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:33.094755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:33.090342 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:33.844293 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:33.844253 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:34.525755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:34.525723 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:34.793868 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:34.793790 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:13:35.247810 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:35.242588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:35.521422 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:35.521337 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:35.694348 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:35.694310 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:36.529748 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:36.512409 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:37.307238 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:37.307193 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:37.913150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:37.913114 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:38.070689 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:38.070604 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:38.246800 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:38.245465 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:38.414705 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:38.414668 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:38.824139 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:38.824102 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:38.992277 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:38.992238 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:39.078223 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:39.078137 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9"] Oct 31 02:13:39.078687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:39.078654 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" containerID="cri-o://b3752f47ecb5cfc5f65a7b6166b624109e270882f555d31492d40c6517e51b2a" gracePeriod=30 Oct 31 02:13:39.089020 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:39.088986 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd"] Oct 31 02:13:39.089245 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:39.089220 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" containerID="cri-o://f62e1d70184d1c7bd3c84a61991b8f7d04ef3a52753d800ccabe0f565d152da8" gracePeriod=30 Oct 31 02:13:39.939381 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:39.939279 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:40.644739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:40.638008 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:41.356743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:41.356563 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:42.046266 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:42.046229 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:42.357444 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:42.357367 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m"] Oct 31 02:13:42.357851 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:42.357576 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" containerID="cri-o://f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b" gracePeriod=30 Oct 31 02:13:42.868985 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:42.868934 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:42.956665 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:42.956624 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" podUID="7bd7c26e-106b-4d86-9885-9026a336feca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:42.956665 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:42.956655 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" podUID="4333d31c-1c31-4394-a273-3fabbc1751ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:13:43.217113 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.217042 5090 patch_prober.go:29] interesting pod/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:13:43.217113 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.217101 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:13:43.692738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.689571 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:43.763099 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.763067 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7"] Oct 31 02:13:43.763331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.763306 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" containerID="cri-o://eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73" gracePeriod=30 Oct 31 02:13:43.784981 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.784949 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq"] Oct 31 02:13:43.785202 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.785172 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" podUID="08527e47-57a3-4a56-a70a-8681a6bce40e" containerName="discovery" containerID="cri-o://16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196" gracePeriod=30 Oct 31 02:13:43.822262 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.822229 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms"] Oct 31 02:13:43.822487 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.822460 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" containerID="cri-o://faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d" gracePeriod=30 Oct 31 02:13:43.877658 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:43.877623 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:44.248368 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.248334 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:13:44.339914 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.339881 5090 generic.go:356] "Generic (PLEG): container finished" podID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerID="f62e1d70184d1c7bd3c84a61991b8f7d04ef3a52753d800ccabe0f565d152da8" exitCode=0 Oct 31 02:13:44.340433 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.340406 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" event={"ID":"429afc9a-4c05-4dac-ba33-e1e25c13ad9c","Type":"ContainerDied","Data":"f62e1d70184d1c7bd3c84a61991b8f7d04ef3a52753d800ccabe0f565d152da8"} Oct 31 02:13:44.342706 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.342680 5090 generic.go:356] "Generic (PLEG): container finished" podID="34c5ace9-0f21-4450-b319-69b5064c4336" containerID="b3752f47ecb5cfc5f65a7b6166b624109e270882f555d31492d40c6517e51b2a" exitCode=0 Oct 31 02:13:44.343047 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.343025 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" event={"ID":"34c5ace9-0f21-4450-b319-69b5064c4336","Type":"ContainerDied","Data":"b3752f47ecb5cfc5f65a7b6166b624109e270882f555d31492d40c6517e51b2a"} Oct 31 02:13:44.344899 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.344875 5090 generic.go:356] "Generic (PLEG): container finished" podID="08527e47-57a3-4a56-a70a-8681a6bce40e" containerID="16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196" exitCode=0 Oct 31 02:13:44.345080 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.345062 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" event={"ID":"08527e47-57a3-4a56-a70a-8681a6bce40e","Type":"ContainerDied","Data":"16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196"} Oct 31 02:13:44.345237 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.345193 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" event={"ID":"08527e47-57a3-4a56-a70a-8681a6bce40e","Type":"ContainerDied","Data":"6a7f23d40f4dcaf6c24a197010aa0395c0d94942dd1ec31ebf96de988764210c"} Oct 31 02:13:44.345365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.345351 5090 scope.go:122] "RemoveContainer" containerID="16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196" Oct 31 02:13:44.345603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.345588 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq" Oct 31 02:13:44.362471 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.362386 5090 scope.go:122] "RemoveContainer" containerID="16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196" Oct 31 02:13:44.362873 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:44.362818 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196\": container with ID starting with 16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196 not found: ID does not exist" containerID="16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196" Oct 31 02:13:44.362873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.362845 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196"} err="failed to get container status \"16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196\": rpc error: code = NotFound desc = could not find container \"16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196\": container with ID starting with 16280210ee39f3759270adf25586656c0a29a9c3bdc61ad2b965e097d550c196 not found: ID does not exist" Oct 31 02:13:44.437297 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.436670 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08527e47-57a3-4a56-a70a-8681a6bce40e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08527e47-57a3-4a56-a70a-8681a6bce40e-local-certs\") pod \"08527e47-57a3-4a56-a70a-8681a6bce40e\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " Oct 31 02:13:44.437297 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.436762 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-ca-configmap\") pod \"08527e47-57a3-4a56-a70a-8681a6bce40e\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " Oct 31 02:13:44.439116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.438077 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-cacerts\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-cacerts\") pod \"08527e47-57a3-4a56-a70a-8681a6bce40e\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " Oct 31 02:13:44.439116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.438119 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-dns-cert\") pod \"08527e47-57a3-4a56-a70a-8681a6bce40e\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " Oct 31 02:13:44.439116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.438221 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-token\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-token\") pod \"08527e47-57a3-4a56-a70a-8681a6bce40e\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " Oct 31 02:13:44.439116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.438251 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-kubeconfig\") pod \"08527e47-57a3-4a56-a70a-8681a6bce40e\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " Oct 31 02:13:44.439116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.438352 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-kube-api-access-rs6w2\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-kube-api-access-rs6w2\") pod \"08527e47-57a3-4a56-a70a-8681a6bce40e\" (UID: \"08527e47-57a3-4a56-a70a-8681a6bce40e\") " Oct 31 02:13:44.440450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.440199 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-ca-configmap" pod "08527e47-57a3-4a56-a70a-8681a6bce40e" (UID: "08527e47-57a3-4a56-a70a-8681a6bce40e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:44.441054 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.441016 5090 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-ca-configmap\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.447671 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.447629 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08527e47-57a3-4a56-a70a-8681a6bce40e-local-certs" pod "08527e47-57a3-4a56-a70a-8681a6bce40e" (UID: "08527e47-57a3-4a56-a70a-8681a6bce40e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.449184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.449148 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-kubeconfig" pod "08527e47-57a3-4a56-a70a-8681a6bce40e" (UID: "08527e47-57a3-4a56-a70a-8681a6bce40e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:44.449288 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.449177 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-dns-cert" pod "08527e47-57a3-4a56-a70a-8681a6bce40e" (UID: "08527e47-57a3-4a56-a70a-8681a6bce40e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:44.449903 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.449874 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-kube-api-access-rs6w2" pod "08527e47-57a3-4a56-a70a-8681a6bce40e" (UID: "08527e47-57a3-4a56-a70a-8681a6bce40e"). InnerVolumeSpecName "kube-api-access-rs6w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:44.450127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.450103 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-token" pod "08527e47-57a3-4a56-a70a-8681a6bce40e" (UID: "08527e47-57a3-4a56-a70a-8681a6bce40e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:44.451311 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.451283 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-cacerts" pod "08527e47-57a3-4a56-a70a-8681a6bce40e" (UID: "08527e47-57a3-4a56-a70a-8681a6bce40e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:13:44.494960 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.494928 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 02:13:44.509985 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.509960 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 02:13:44.542247 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.542218 5090 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/08527e47-57a3-4a56-a70a-8681a6bce40e-local-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.542247 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.542246 5090 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-cacerts\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.542420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.542257 5090 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-csr-dns-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.542420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.542268 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.542420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.542277 5090 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/08527e47-57a3-4a56-a70a-8681a6bce40e-istio-kubeconfig\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.542420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.542285 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6w2\" (UniqueName: \"kubernetes.io/projected/08527e47-57a3-4a56-a70a-8681a6bce40e-kube-api-access-rs6w2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.571193 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.571160 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:44.643655 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643582 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-podinfo\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.643655 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643625 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-data\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.643863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643698 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-token\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-token\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.643863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643759 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-istio-token\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-istio-token\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.643863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643812 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-certs\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.643863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643850 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c5ace9-0f21-4450-b319-69b5064c4336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5ace9-0f21-4450-b319-69b5064c4336-istiod-ca-cert\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.644061 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643901 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-kube-api-access-th9xs\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-kube-api-access-th9xs\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.644061 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643996 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-socket\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.644176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644059 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34c5ace9-0f21-4450-b319-69b5064c4336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5ace9-0f21-4450-b319-69b5064c4336-istio-podinfo\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.644176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644108 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-envoy\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.644176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.643993 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-data" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.644176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644120 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-certs" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.644391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644368 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-credential-socket" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.644391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644378 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-socket" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.644503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644144 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-credential-socket\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.644503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644410 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c5ace9-0f21-4450-b319-69b5064c4336-istiod-ca-cert" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:44.644503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644472 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-envoy\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.644678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644527 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-certs\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.644678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644559 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-data\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644766 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-kube-api-access-qrfr2\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-kube-api-access-qrfr2\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644816 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-socket\") pod \"34c5ace9-0f21-4450-b319-69b5064c4336\" (UID: \"34c5ace9-0f21-4450-b319-69b5064c4336\") " Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644925 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-credential-socket\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.644954 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istiod-ca-cert\") pod \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\" (UID: \"429afc9a-4c05-4dac-ba33-e1e25c13ad9c\") " Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.645055 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-certs" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.645254 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-socket" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.645509 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-data" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.645573 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-credential-socket" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.645906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.645620 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istiod-ca-cert" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646008 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646032 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646056 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646071 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646086 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34c5ace9-0f21-4450-b319-69b5064c4336-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646102 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646119 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646132 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646146 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.646489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646161 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.647151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646785 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-istio-token" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:44.647151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646914 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-kube-api-access-th9xs" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "kube-api-access-th9xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:44.647151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.646951 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-token" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:44.647335 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.647206 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34c5ace9-0f21-4450-b319-69b5064c4336-istio-podinfo" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:44.647455 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.647434 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-podinfo" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:44.647792 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.647771 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-envoy" pod "34c5ace9-0f21-4450-b319-69b5064c4336" (UID: "34c5ace9-0f21-4450-b319-69b5064c4336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.649205 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.649169 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-envoy" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:44.649424 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.649406 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-kube-api-access-qrfr2" pod "429afc9a-4c05-4dac-ba33-e1e25c13ad9c" (UID: "429afc9a-4c05-4dac-ba33-e1e25c13ad9c"). InnerVolumeSpecName "kube-api-access-qrfr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:44.694760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.693519 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq"] Oct 31 02:13:44.702865 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.702787 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k8mmq"] Oct 31 02:13:44.747480 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747448 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.747480 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747476 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.747686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747489 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th9xs\" (UniqueName: \"kubernetes.io/projected/34c5ace9-0f21-4450-b319-69b5064c4336-kube-api-access-th9xs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.747686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747498 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34c5ace9-0f21-4450-b319-69b5064c4336-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.747686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747507 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34c5ace9-0f21-4450-b319-69b5064c4336-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.747686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747517 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.747686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747525 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrfr2\" (UniqueName: \"kubernetes.io/projected/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-kube-api-access-qrfr2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:44.747686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:44.747534 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/429afc9a-4c05-4dac-ba33-e1e25c13ad9c-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:45.347503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.347470 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:45.401984 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.401951 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" Oct 31 02:13:45.402177 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.401954 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9" event={"ID":"34c5ace9-0f21-4450-b319-69b5064c4336","Type":"ContainerDied","Data":"27654ac3606b17d7a22867cb2a534a426e08c8fc29d62a67466d70e5ede381c0"} Oct 31 02:13:45.402177 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.402119 5090 scope.go:122] "RemoveContainer" containerID="b3752f47ecb5cfc5f65a7b6166b624109e270882f555d31492d40c6517e51b2a" Oct 31 02:13:45.407333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.407309 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" event={"ID":"429afc9a-4c05-4dac-ba33-e1e25c13ad9c","Type":"ContainerDied","Data":"f60d9f12e3acf18a7ab9327b0581acfadfeece6bfaba2e49a2845938ccb5fff9"} Oct 31 02:13:45.407490 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.407313 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd" Oct 31 02:13:45.415295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.415269 5090 scope.go:122] "RemoveContainer" containerID="f62e1d70184d1c7bd3c84a61991b8f7d04ef3a52753d800ccabe0f565d152da8" Oct 31 02:13:45.434366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.432183 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9"] Oct 31 02:13:45.436059 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.436034 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-ht4k9"] Oct 31 02:13:45.449260 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.449234 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd"] Oct 31 02:13:45.455846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.455823 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp"] Oct 31 02:13:45.456055 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.456034 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" containerID="cri-o://2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85" gracePeriod=30 Oct 31 02:13:45.458760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.458734 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-9ldvd"] Oct 31 02:13:45.748249 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.748218 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08527e47-57a3-4a56-a70a-8681a6bce40e" path="/var/lib/kubelet/pods/08527e47-57a3-4a56-a70a-8681a6bce40e/volumes" Oct 31 02:13:45.748766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.748751 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" path="/var/lib/kubelet/pods/34c5ace9-0f21-4450-b319-69b5064c4336/volumes" Oct 31 02:13:45.749194 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:45.749181 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" path="/var/lib/kubelet/pods/429afc9a-4c05-4dac-ba33-e1e25c13ad9c/volumes" Oct 31 02:13:46.142385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.142357 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:46.159902 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.156969 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 02:13:46.248765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.244442 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 02:13:46.280269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.280194 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd7c26e-106b-4d86-9885-9026a336feca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c26e-106b-4d86-9885-9026a336feca-plugins-config-volume\") pod \"7bd7c26e-106b-4d86-9885-9026a336feca\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " Oct 31 02:13:46.280437 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.280419 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd7c26e-106b-4d86-9885-9026a336feca-kube-api-access-bqtqd\" (UniqueName: \"kubernetes.io/projected/7bd7c26e-106b-4d86-9885-9026a336feca-kube-api-access-bqtqd\") pod \"7bd7c26e-106b-4d86-9885-9026a336feca\" (UID: \"7bd7c26e-106b-4d86-9885-9026a336feca\") " Oct 31 02:13:46.280636 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.280614 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd7c26e-106b-4d86-9885-9026a336feca-plugins-config-volume" pod "7bd7c26e-106b-4d86-9885-9026a336feca" (UID: "7bd7c26e-106b-4d86-9885-9026a336feca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:46.280955 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.280934 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bd7c26e-106b-4d86-9885-9026a336feca-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:46.282920 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.282890 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd7c26e-106b-4d86-9885-9026a336feca-kube-api-access-bqtqd" pod "7bd7c26e-106b-4d86-9885-9026a336feca" (UID: "7bd7c26e-106b-4d86-9885-9026a336feca"). InnerVolumeSpecName "kube-api-access-bqtqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:46.382268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.382239 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4333d31c-1c31-4394-a273-3fabbc1751ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4333d31c-1c31-4394-a273-3fabbc1751ca-plugins-config-volume\") pod \"4333d31c-1c31-4394-a273-3fabbc1751ca\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " Oct 31 02:13:46.382423 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.382288 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4333d31c-1c31-4394-a273-3fabbc1751ca-kube-api-access-9rbtq\" (UniqueName: \"kubernetes.io/projected/4333d31c-1c31-4394-a273-3fabbc1751ca-kube-api-access-9rbtq\") pod \"4333d31c-1c31-4394-a273-3fabbc1751ca\" (UID: \"4333d31c-1c31-4394-a273-3fabbc1751ca\") " Oct 31 02:13:46.382618 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.382598 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4333d31c-1c31-4394-a273-3fabbc1751ca-plugins-config-volume" pod "4333d31c-1c31-4394-a273-3fabbc1751ca" (UID: "4333d31c-1c31-4394-a273-3fabbc1751ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:46.382659 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.382645 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqtqd\" (UniqueName: \"kubernetes.io/projected/7bd7c26e-106b-4d86-9885-9026a336feca-kube-api-access-bqtqd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:46.384541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.384522 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4333d31c-1c31-4394-a273-3fabbc1751ca-kube-api-access-9rbtq" pod "4333d31c-1c31-4394-a273-3fabbc1751ca" (UID: "4333d31c-1c31-4394-a273-3fabbc1751ca"). InnerVolumeSpecName "kube-api-access-9rbtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:46.415393 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.415361 5090 generic.go:356] "Generic (PLEG): container finished" podID="7bd7c26e-106b-4d86-9885-9026a336feca" containerID="f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0" exitCode=0 Oct 31 02:13:46.415540 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.415414 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" Oct 31 02:13:46.415540 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.415451 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" event={"ID":"7bd7c26e-106b-4d86-9885-9026a336feca","Type":"ContainerDied","Data":"f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0"} Oct 31 02:13:46.415540 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.415503 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9" event={"ID":"7bd7c26e-106b-4d86-9885-9026a336feca","Type":"ContainerDied","Data":"8f298bfaaffbdee1a7cdd94f948d1cf8763eabfcea8341a918daf25c4db18934"} Oct 31 02:13:46.415540 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.415532 5090 scope.go:122] "RemoveContainer" containerID="f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0" Oct 31 02:13:46.416922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.416825 5090 generic.go:356] "Generic (PLEG): container finished" podID="4333d31c-1c31-4394-a273-3fabbc1751ca" containerID="7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9" exitCode=0 Oct 31 02:13:46.416922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.416878 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" Oct 31 02:13:46.417056 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.416917 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" event={"ID":"4333d31c-1c31-4394-a273-3fabbc1751ca","Type":"ContainerDied","Data":"7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9"} Oct 31 02:13:46.417056 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.416974 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv" event={"ID":"4333d31c-1c31-4394-a273-3fabbc1751ca","Type":"ContainerDied","Data":"d1f850793aedee9338a1e845e7c3439bcfd573940f86357f4ff95891a17bb663"} Oct 31 02:13:46.423867 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.423841 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:13:46.424385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.423892 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:13:46.427560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.427539 5090 scope.go:122] "RemoveContainer" containerID="f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0" Oct 31 02:13:46.427869 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:46.427844 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0\": container with ID starting with f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0 not found: ID does not exist" containerID="f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0" Oct 31 02:13:46.427946 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.427881 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0"} err="failed to get container status \"f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0\": rpc error: code = NotFound desc = could not find container \"f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0\": container with ID starting with f0b4165d3f20436285c82b06eebf95c9cd9f1929024d0fbf428686e5e7b69af0 not found: ID does not exist" Oct 31 02:13:46.427946 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.427903 5090 scope.go:122] "RemoveContainer" containerID="7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9" Oct 31 02:13:46.437433 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.437414 5090 scope.go:122] "RemoveContainer" containerID="7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9" Oct 31 02:13:46.437698 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:46.437681 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9\": container with ID starting with 7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9 not found: ID does not exist" containerID="7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9" Oct 31 02:13:46.437830 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.437709 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9"} err="failed to get container status \"7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9\": rpc error: code = NotFound desc = could not find container \"7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9\": container with ID starting with 7710d68ded2c9ce9ddd02ce4ccd903d5e379949b3586452149301c415770f1d9 not found: ID does not exist" Oct 31 02:13:46.442493 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.442473 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9"] Oct 31 02:13:46.449752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.449727 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-6c54a1-67c898496c-nt8g9"] Oct 31 02:13:46.460125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.460095 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv"] Oct 31 02:13:46.469136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.469113 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-6c54a1-747c59c99f-2n9cv"] Oct 31 02:13:46.484338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.484307 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4333d31c-1c31-4394-a273-3fabbc1751ca-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:46.484338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.484335 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rbtq\" (UniqueName: \"kubernetes.io/projected/4333d31c-1c31-4394-a273-3fabbc1751ca-kube-api-access-9rbtq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:46.952250 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:46.952214 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:47.634816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.634783 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:47.671564 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.671538 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 02:13:47.751660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.751625 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4333d31c-1c31-4394-a273-3fabbc1751ca" path="/var/lib/kubelet/pods/4333d31c-1c31-4394-a273-3fabbc1751ca/volumes" Oct 31 02:13:47.752330 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.752293 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd7c26e-106b-4d86-9885-9026a336feca" path="/var/lib/kubelet/pods/7bd7c26e-106b-4d86-9885-9026a336feca/volumes" Oct 31 02:13:47.805283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805255 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d090668-ad32-413e-9768-360d58a3a7cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d090668-ad32-413e-9768-360d58a3a7cc-istiod-ca-cert\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805423 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805402 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-envoy\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805495 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805441 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-kube-api-access-264n9\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-kube-api-access-264n9\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805495 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805472 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-data\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805622 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805562 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-credential-socket\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805618 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-certs\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805652 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d090668-ad32-413e-9768-360d58a3a7cc-istiod-ca-cert" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:47.805680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805670 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-istio-token\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-istio-token\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805741 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-socket\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.805877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.805814 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d090668-ad32-413e-9768-360d58a3a7cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d090668-ad32-413e-9768-360d58a3a7cc-istio-podinfo\") pod \"0d090668-ad32-413e-9768-360d58a3a7cc\" (UID: \"0d090668-ad32-413e-9768-360d58a3a7cc\") " Oct 31 02:13:47.806058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.806027 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-credential-socket" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:47.806138 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.806068 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-certs" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:47.807109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.806330 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-socket" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:47.807109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.806852 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.807109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.806876 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.807109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.806891 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.807109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.806906 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d090668-ad32-413e-9768-360d58a3a7cc-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.807452 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.807173 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-data" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:47.809217 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.809187 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-istio-token" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:47.809322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.809256 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d090668-ad32-413e-9768-360d58a3a7cc-istio-podinfo" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:47.809439 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.809411 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-envoy" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:47.809709 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.809692 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-kube-api-access-264n9" pod "0d090668-ad32-413e-9768-360d58a3a7cc" (UID: "0d090668-ad32-413e-9768-360d58a3a7cc"). InnerVolumeSpecName "kube-api-access-264n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:47.907944 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.907866 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.907944 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.907897 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d090668-ad32-413e-9768-360d58a3a7cc-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.907944 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.907907 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.907944 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.907916 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264n9\" (UniqueName: \"kubernetes.io/projected/0d090668-ad32-413e-9768-360d58a3a7cc-kube-api-access-264n9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.907944 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.907927 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d090668-ad32-413e-9768-360d58a3a7cc-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:47.936780 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.936748 5090 patch_prober.go:29] interesting pod/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:13:47.936938 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:47.936804 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:13:48.440985 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.440938 5090 generic.go:356] "Generic (PLEG): container finished" podID="0d090668-ad32-413e-9768-360d58a3a7cc" containerID="f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b" exitCode=0 Oct 31 02:13:48.441478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.441020 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" event={"ID":"0d090668-ad32-413e-9768-360d58a3a7cc","Type":"ContainerDied","Data":"f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b"} Oct 31 02:13:48.441478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.441084 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" event={"ID":"0d090668-ad32-413e-9768-360d58a3a7cc","Type":"ContainerDied","Data":"5cce4ed46734719434ac18a7741adad2ebfa37961be7c358e4ce66c72020900f"} Oct 31 02:13:48.441478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.441111 5090 scope.go:122] "RemoveContainer" containerID="f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b" Oct 31 02:13:48.441478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.440991 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m" Oct 31 02:13:48.446965 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.446937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:48.455427 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.455407 5090 scope.go:122] "RemoveContainer" containerID="f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b" Oct 31 02:13:48.455786 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:48.455763 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b\": container with ID starting with f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b not found: ID does not exist" containerID="f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b" Oct 31 02:13:48.455886 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.455795 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b"} err="failed to get container status \"f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b\": rpc error: code = NotFound desc = could not find container \"f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b\": container with ID starting with f7004d16b428552944fcf064535c63061b528de66d543e7fc18abc2e70a40c1b not found: ID does not exist" Oct 31 02:13:48.472179 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.472154 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m"] Oct 31 02:13:48.479613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.479591 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-4xs9m"] Oct 31 02:13:48.501531 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.501504 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f"] Oct 31 02:13:48.502257 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502240 5090 cpu_manager.go:414] "containerMap: removing container" podUID="7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" containerName="maas-api" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502261 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" containerName="maas-api" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502272 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502277 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502292 5090 cpu_manager.go:414] "containerMap: removing container" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502298 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502305 5090 cpu_manager.go:414] "containerMap: removing container" podUID="4333d31c-1c31-4394-a273-3fabbc1751ca" containerName="payload-pre-processing" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502310 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4333d31c-1c31-4394-a273-3fabbc1751ca" containerName="payload-pre-processing" Oct 31 02:13:48.502325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502326 5090 cpu_manager.go:414] "containerMap: removing container" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502331 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502339 5090 cpu_manager.go:414] "containerMap: removing container" podUID="3aff8334-9229-4a72-89b7-5128f0d59380" containerName="maas-api" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502344 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aff8334-9229-4a72-89b7-5128f0d59380" containerName="maas-api" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502358 5090 cpu_manager.go:414] "containerMap: removing container" podUID="f4dff408-6802-4435-a521-cbd7d32eb7d4" containerName="payload-processing" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502362 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dff408-6802-4435-a521-cbd7d32eb7d4" containerName="payload-processing" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502375 5090 cpu_manager.go:414] "containerMap: removing container" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502380 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502386 5090 cpu_manager.go:414] "containerMap: removing container" podUID="7bd7c26e-106b-4d86-9885-9026a336feca" containerName="payload-processing" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502391 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd7c26e-106b-4d86-9885-9026a336feca" containerName="payload-processing" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502409 5090 cpu_manager.go:414] "containerMap: removing container" podUID="08527e47-57a3-4a56-a70a-8681a6bce40e" containerName="discovery" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502413 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08527e47-57a3-4a56-a70a-8681a6bce40e" containerName="discovery" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502496 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dff408-6802-4435-a521-cbd7d32eb7d4" containerName="payload-processing" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502508 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="429afc9a-4c05-4dac-ba33-e1e25c13ad9c" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502514 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd7c26e-106b-4d86-9885-9026a336feca" containerName="payload-processing" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502521 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="08527e47-57a3-4a56-a70a-8681a6bce40e" containerName="discovery" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502529 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3b3d41-864b-490f-a5e1-b2ca1cff3b1b" containerName="maas-api" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502535 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20804cd-2d74-4ca2-b037-d8ff00e003d4" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502542 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c5ace9-0f21-4450-b319-69b5064c4336" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502548 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="4333d31c-1c31-4394-a273-3fabbc1751ca" containerName="payload-pre-processing" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502556 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" containerName="istio-proxy" Oct 31 02:13:48.502597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.502561 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aff8334-9229-4a72-89b7-5128f0d59380" containerName="maas-api" Oct 31 02:13:48.508406 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.508381 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.570740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.556786 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f"] Oct 31 02:13:48.591830 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.563763 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:13:48.592133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.592100 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:13:48.619766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.617932 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-routing-overlay-volume\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.619766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.617996 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-provider-credentials\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.619766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.618058 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-plugins-config-volume\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.619766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.618284 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl6dv\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-kube-api-access-gl6dv\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.728397 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.725115 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl6dv\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-kube-api-access-gl6dv\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.732732 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.732683 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-routing-overlay-volume\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.744823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.734195 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-routing-overlay-volume\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.745106 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.744026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:48.745293 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.744808 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-provider-credentials\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.745380 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.745362 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-plugins-config-volume\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.746489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.746404 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-plugins-config-volume\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.764807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.764775 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl6dv\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-kube-api-access-gl6dv\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.806774 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.803686 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-provider-credentials\") pod \"payload-processing-external-model-5fd8fb4945-wsw8f\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.828085 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:48.828058 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:48.847612 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:48.847572 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-dd41e9644f67: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:48.847612 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:48.847602 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:48.847829 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:48.847676 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials podName:529cc133-aef7-4fa5-872b-c6e1713742be nodeName:}" failed. No retries permitted until 2026-10-31 02:13:49.347655382 +0000 UTC m=+3229.727900606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be") : secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:49.111752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.104774 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79"] Oct 31 02:13:49.161971 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:13:49.161930 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b85e89_5c51_46ac_b7e8_7128e4d773bf.slice/crio-39bc2212e0da7967b0b33a675dc344e06e9cee5ca22f7459f5b1b8c373ebc236 WatchSource:0}: Error finding container 39bc2212e0da7967b0b33a675dc344e06e9cee5ca22f7459f5b1b8c373ebc236: Status 404 returned error can't find the container with id 39bc2212e0da7967b0b33a675dc344e06e9cee5ca22f7459f5b1b8c373ebc236 Oct 31 02:13:49.172924 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.172890 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f"] Oct 31 02:13:49.284396 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.284371 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 02:13:49.372198 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.371842 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 02:13:49.382485 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382456 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-data\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382612 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382594 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-socket\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382651 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382631 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-istio-token\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-istio-token\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382684 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382658 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-credential-socket\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382732 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-data" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.382806 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382757 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-certs\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382840 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382821 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-kube-api-access-998tr\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-kube-api-access-998tr\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382890 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6075252e-be6b-444f-99d6-68292c59f906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6075252e-be6b-444f-99d6-68292c59f906-istio-podinfo\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382943 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382931 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6075252e-be6b-444f-99d6-68292c59f906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6075252e-be6b-444f-99d6-68292c59f906-istiod-ca-cert\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.382989 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382974 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-envoy\") pod \"6075252e-be6b-444f-99d6-68292c59f906\" (UID: \"6075252e-be6b-444f-99d6-68292c59f906\") " Oct 31 02:13:49.383035 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.382973 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-credential-socket" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.383275 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6075252e-be6b-444f-99d6-68292c59f906-istiod-ca-cert" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.383391 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-certs" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.383669 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.383692 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.383708 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.383761 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6075252e-be6b-444f-99d6-68292c59f906-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:49.383861 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-dd41e9644f67: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:49.383874 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:49.384230 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:49.383931 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials podName:529cc133-aef7-4fa5-872b-c6e1713742be nodeName:}" failed. No retries permitted until 2026-10-31 02:13:50.383911492 +0000 UTC m=+3230.764156710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be") : secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:49.386093 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.386034 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-socket" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.387646 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.387586 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-istio-token" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:49.388551 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.388518 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-kube-api-access-998tr" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "kube-api-access-998tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:49.389692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.389671 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-envoy" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.390765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.390230 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6075252e-be6b-444f-99d6-68292c59f906-istio-podinfo" pod "6075252e-be6b-444f-99d6-68292c59f906" (UID: "6075252e-be6b-444f-99d6-68292c59f906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:49.451762 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.451704 5090 generic.go:356] "Generic (PLEG): container finished" podID="6075252e-be6b-444f-99d6-68292c59f906" containerID="eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73" exitCode=0 Oct 31 02:13:49.452385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.451835 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" event={"ID":"6075252e-be6b-444f-99d6-68292c59f906","Type":"ContainerDied","Data":"eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73"} Oct 31 02:13:49.452539 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.452522 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" event={"ID":"6075252e-be6b-444f-99d6-68292c59f906","Type":"ContainerDied","Data":"320152e5f6307d11d37cb53021d15ca3843aafd68a952d3b0a906f7257e8160f"} Oct 31 02:13:49.452639 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.451873 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7" Oct 31 02:13:49.452934 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.452565 5090 scope.go:122] "RemoveContainer" containerID="eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73" Oct 31 02:13:49.458484 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.457400 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" event={"ID":"e6b85e89-5c51-46ac-b7e8-7128e4d773bf","Type":"ContainerStarted","Data":"2d301d7830b1901a175207219116c196ceead62dd3c70ec69d8974aa5e2824a5"} Oct 31 02:13:49.458484 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.457459 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" event={"ID":"e6b85e89-5c51-46ac-b7e8-7128e4d773bf","Type":"ContainerStarted","Data":"39bc2212e0da7967b0b33a675dc344e06e9cee5ca22f7459f5b1b8c373ebc236"} Oct 31 02:13:49.458484 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.458426 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:13:49.468420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.468397 5090 generic.go:356] "Generic (PLEG): container finished" podID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerID="faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d" exitCode=0 Oct 31 02:13:49.468600 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.468584 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" Oct 31 02:13:49.468741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.468705 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" event={"ID":"3c35804e-37df-4f7a-b0c6-2eb266f31523","Type":"ContainerDied","Data":"faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d"} Oct 31 02:13:49.468822 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.468764 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms" event={"ID":"3c35804e-37df-4f7a-b0c6-2eb266f31523","Type":"ContainerDied","Data":"cf16a13c91aecb92c681b179d50ba7dc4bbfa31f0546d615a1aeaeee7c982d7e"} Oct 31 02:13:49.473374 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.473354 5090 scope.go:122] "RemoveContainer" containerID="eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73" Oct 31 02:13:49.473662 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:49.473642 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73\": container with ID starting with eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73 not found: ID does not exist" containerID="eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73" Oct 31 02:13:49.473740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.473669 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73"} err="failed to get container status \"eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73\": rpc error: code = NotFound desc = could not find container \"eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73\": container with ID starting with eb8e73e5c15d65d598380ed9a48ee58cf3b204c4f2d858d317f297d7f833bb73 not found: ID does not exist" Oct 31 02:13:49.473740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.473685 5090 scope.go:122] "RemoveContainer" containerID="faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d" Oct 31 02:13:49.484429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484374 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" podStartSLOduration=1.484359282 podStartE2EDuration="1.484359282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:49.481087609 +0000 UTC m=+3229.861332833" watchObservedRunningTime="2026-10-31 02:13:49.484359282 +0000 UTC m=+3229.864604514" Oct 31 02:13:49.484544 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484431 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-socket\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.484544 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484478 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-data\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.484685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484568 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-token\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-token\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.484685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484620 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-podinfo\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.484943 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484924 5090 scope.go:122] "RemoveContainer" containerID="faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d" Oct 31 02:13:49.485002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484953 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c35804e-37df-4f7a-b0c6-2eb266f31523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35804e-37df-4f7a-b0c6-2eb266f31523-istiod-ca-cert\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.485002 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.484973 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-data" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.485115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485003 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-envoy\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.485115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485045 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-kube-api-access-brwv7\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-kube-api-access-brwv7\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.485115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485075 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-certs\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.485115 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485105 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-credential-socket\") pod \"3c35804e-37df-4f7a-b0c6-2eb266f31523\" (UID: \"3c35804e-37df-4f7a-b0c6-2eb266f31523\") " Oct 31 02:13:49.485328 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:49.485209 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d\": container with ID starting with faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d not found: ID does not exist" containerID="faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d" Oct 31 02:13:49.485328 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485239 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d"} err="failed to get container status \"faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d\": rpc error: code = NotFound desc = could not find container \"faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d\": container with ID starting with faadccb46ec69458cbcd06b29fbf1f6915279f88aefe50fe472753caf899222d not found: ID does not exist" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485659 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c35804e-37df-4f7a-b0c6-2eb266f31523-istiod-ca-cert" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485704 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-socket" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485736 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485755 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485772 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-998tr\" (UniqueName: \"kubernetes.io/projected/6075252e-be6b-444f-99d6-68292c59f906-kube-api-access-998tr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485787 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485800 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6075252e-be6b-444f-99d6-68292c59f906-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.485687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485817 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6075252e-be6b-444f-99d6-68292c59f906-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.486379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.485983 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-certs" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.486379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.486029 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-credential-socket" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.487483 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.487458 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-kube-api-access-brwv7" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "kube-api-access-brwv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:49.487552 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.487482 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-token" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:49.487658 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.487643 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-podinfo" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:49.487870 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.487856 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-envoy" pod "3c35804e-37df-4f7a-b0c6-2eb266f31523" (UID: "3c35804e-37df-4f7a-b0c6-2eb266f31523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:49.496906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.496888 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:49.514743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.514597 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7"] Oct 31 02:13:49.521877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.521855 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-ch8d7"] Oct 31 02:13:49.586896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586860 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwv7\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-kube-api-access-brwv7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.586896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586892 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.587158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586905 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.587158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586920 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.587158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586933 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.587158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586947 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.587158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586960 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c35804e-37df-4f7a-b0c6-2eb266f31523-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.587158 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.586975 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c35804e-37df-4f7a-b0c6-2eb266f31523-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:49.639416 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.639344 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:49.748483 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.748440 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d090668-ad32-413e-9768-360d58a3a7cc" path="/var/lib/kubelet/pods/0d090668-ad32-413e-9768-360d58a3a7cc/volumes" Oct 31 02:13:49.748980 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.748960 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6075252e-be6b-444f-99d6-68292c59f906" path="/var/lib/kubelet/pods/6075252e-be6b-444f-99d6-68292c59f906/volumes" Oct 31 02:13:49.778370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.778340 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:49.786029 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.786007 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms"] Oct 31 02:13:49.789549 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:49.789521 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-9xmms"] Oct 31 02:13:50.258113 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.258029 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:50.403263 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.403225 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:50.482565 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:50.482536 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-dd41e9644f67: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:50.482565 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:50.482559 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:50.483005 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:50.482613 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials podName:529cc133-aef7-4fa5-872b-c6e1713742be nodeName:}" failed. No retries permitted until 2026-10-31 02:13:52.482598161 +0000 UTC m=+3232.862843378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be") : secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:50.556369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.556289 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:50.708883 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.708844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:50.762121 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.762093 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 02:13:50.787200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787155 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-data\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787214 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-certs\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787279 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-kube-api-access-995px\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-kube-api-access-995px\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787301 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-token\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-token\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787368 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-podinfo\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787539 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e2b35e-3e13-4c16-8703-21ff1860286f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e2b35e-3e13-4c16-8703-21ff1860286f-istiod-ca-cert\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787601 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-socket\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787647 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-envoy\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.787765 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787675 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-credential-socket\") pod \"63e2b35e-3e13-4c16-8703-21ff1860286f\" (UID: \"63e2b35e-3e13-4c16-8703-21ff1860286f\") " Oct 31 02:13:50.788030 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.787957 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-data" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:50.788359 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.788334 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.789607 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.789085 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-certs" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:50.789607 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.789135 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e2b35e-3e13-4c16-8703-21ff1860286f-istiod-ca-cert" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:50.789607 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.789280 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-socket" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:50.789607 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.789577 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-credential-socket" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:50.790941 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.790913 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-token" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:50.791537 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.791515 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-kube-api-access-995px" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "kube-api-access-995px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:50.791996 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.791959 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-podinfo" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:13:50.791996 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.791974 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-envoy" pod "63e2b35e-3e13-4c16-8703-21ff1860286f" (UID: "63e2b35e-3e13-4c16-8703-21ff1860286f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890296 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63e2b35e-3e13-4c16-8703-21ff1860286f-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890330 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890345 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890357 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890365 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63e2b35e-3e13-4c16-8703-21ff1860286f-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890374 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-995px\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-kube-api-access-995px\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890383 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.891896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.890392 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63e2b35e-3e13-4c16-8703-21ff1860286f-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:50.897826 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:50.897795 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:51.347540 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.347506 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:51.488796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.488758 5090 generic.go:356] "Generic (PLEG): container finished" podID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerID="2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85" exitCode=0 Oct 31 02:13:51.489269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.488815 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" Oct 31 02:13:51.489269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.488864 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" event={"ID":"63e2b35e-3e13-4c16-8703-21ff1860286f","Type":"ContainerDied","Data":"2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85"} Oct 31 02:13:51.489269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.488945 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp" event={"ID":"63e2b35e-3e13-4c16-8703-21ff1860286f","Type":"ContainerDied","Data":"a70cc7c293c04b7ac53c6ebf7ffc8b8b999ec0e7574166ac7ac0ef61ff78cd05"} Oct 31 02:13:51.489269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.488977 5090 scope.go:122] "RemoveContainer" containerID="2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85" Oct 31 02:13:51.496262 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.496236 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:51.504595 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.504571 5090 scope.go:122] "RemoveContainer" containerID="2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85" Oct 31 02:13:51.505044 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:51.504972 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85\": container with ID starting with 2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85 not found: ID does not exist" containerID="2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85" Oct 31 02:13:51.505044 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.505009 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85"} err="failed to get container status \"2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85\": rpc error: code = NotFound desc = could not find container \"2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85\": container with ID starting with 2b4fcfc44312aa1d3068cacae4e52a9ca7555e82ed04f596df6a09d8c7240c85 not found: ID does not exist" Oct 31 02:13:51.516881 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.516858 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp"] Oct 31 02:13:51.521680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.521658 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-6fzbp"] Oct 31 02:13:51.750967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.750935 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" path="/var/lib/kubelet/pods/3c35804e-37df-4f7a-b0c6-2eb266f31523/volumes" Oct 31 02:13:51.751605 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:51.751586 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" path="/var/lib/kubelet/pods/63e2b35e-3e13-4c16-8703-21ff1860286f/volumes" Oct 31 02:13:52.192308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.192214 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:52.339372 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.339336 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5"] Oct 31 02:13:52.339994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.339979 5090 cpu_manager.go:414] "containerMap: removing container" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" Oct 31 02:13:52.340042 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.339996 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" Oct 31 02:13:52.340042 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.340010 5090 cpu_manager.go:414] "containerMap: removing container" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" Oct 31 02:13:52.340042 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.340016 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" Oct 31 02:13:52.340150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.340043 5090 cpu_manager.go:414] "containerMap: removing container" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" Oct 31 02:13:52.340150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.340048 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" Oct 31 02:13:52.340150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.340133 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e2b35e-3e13-4c16-8703-21ff1860286f" containerName="istio-proxy" Oct 31 02:13:52.340150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.340146 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c35804e-37df-4f7a-b0c6-2eb266f31523" containerName="istio-proxy" Oct 31 02:13:52.340268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.340153 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="6075252e-be6b-444f-99d6-68292c59f906" containerName="istio-proxy" Oct 31 02:13:52.345195 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.345174 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.346368 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.346346 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:52.354667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.354303 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"agc-cred-7e7639d4a803\"" Oct 31 02:13:52.354667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.354440 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-e2e--43356fff-dockercfg-hmp2f\"" Oct 31 02:13:52.354667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.354506 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"kube-root-ca.crt\"" Oct 31 02:13:52.354667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.354584 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"routing-overlay\"" Oct 31 02:13:52.354667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.354626 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bb5c51\"" Oct 31 02:13:52.355001 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.354702 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bb5c51\"/\"openshift-service-ca.crt\"" Oct 31 02:13:52.355391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.355368 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5"] Oct 31 02:13:52.431086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.431043 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.431291 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.431098 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.431360 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.431327 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xb6\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-kube-api-access-c9xb6\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.431553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.431533 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.533236 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.533191 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xb6\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-kube-api-access-c9xb6\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.533758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.533349 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.533758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.533396 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.533758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.533422 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.533758 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:52.533589 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0b6abf/agc-cred-dd41e9644f67: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:52.533758 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:52.533604 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79: secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:52.533758 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:52.533657 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials podName:529cc133-aef7-4fa5-872b-c6e1713742be nodeName:}" failed. No retries permitted until 2026-10-31 02:13:56.53363984 +0000 UTC m=+3236.913885058 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be") : secret "agc-cred-dd41e9644f67" not found Oct 31 02:13:52.534503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.534474 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.534503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.534489 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.537064 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.537040 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.545272 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.545246 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xb6\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-kube-api-access-c9xb6\") pod \"payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.662632 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.662591 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:52.958086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:52.958052 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5"] Oct 31 02:13:52.959555 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:13:52.959525 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3968f3a1_4bc1_4e90_a4b2_6d1542d6353b.slice/crio-46a98f5cde2963d8fc6b6b62280178f7d756e6d0a54efb2263c5e38dce6de812 WatchSource:0}: Error finding container 46a98f5cde2963d8fc6b6b62280178f7d756e6d0a54efb2263c5e38dce6de812: Status 404 returned error can't find the container with id 46a98f5cde2963d8fc6b6b62280178f7d756e6d0a54efb2263c5e38dce6de812 Oct 31 02:13:53.044474 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:53.044440 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:53.511649 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:53.511611 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" event={"ID":"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b","Type":"ContainerStarted","Data":"1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d"} Oct 31 02:13:53.511884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:53.511666 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" event={"ID":"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b","Type":"ContainerStarted","Data":"46a98f5cde2963d8fc6b6b62280178f7d756e6d0a54efb2263c5e38dce6de812"} Oct 31 02:13:53.511884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:53.511865 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:13:53.533011 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:53.532958 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" podStartSLOduration=1.532942712 podStartE2EDuration="1.532942712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:13:53.529616059 +0000 UTC m=+3233.909861294" watchObservedRunningTime="2026-10-31 02:13:53.532942712 +0000 UTC m=+3233.913187934" Oct 31 02:13:53.767530 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:53.767407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:54.367790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.367754 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:54.469063 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.469020 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" podUID="529cc133-aef7-4fa5-872b-c6e1713742be" containerName="payload-processing" containerID="cri-o://8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74" gracePeriod=25 Oct 31 02:13:54.738103 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.738075 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:13:54.897456 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.897369 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-kube-api-access-5vtn2\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-kube-api-access-5vtn2\") pod \"529cc133-aef7-4fa5-872b-c6e1713742be\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " Oct 31 02:13:54.897998 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.897473 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials\") pod \"529cc133-aef7-4fa5-872b-c6e1713742be\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " Oct 31 02:13:54.897998 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.897498 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-plugins-config-volume\") pod \"529cc133-aef7-4fa5-872b-c6e1713742be\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " Oct 31 02:13:54.897998 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.897597 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-routing-overlay-volume\") pod \"529cc133-aef7-4fa5-872b-c6e1713742be\" (UID: \"529cc133-aef7-4fa5-872b-c6e1713742be\") " Oct 31 02:13:54.898188 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.898052 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-plugins-config-volume" pod "529cc133-aef7-4fa5-872b-c6e1713742be" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:54.898188 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.898065 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-routing-overlay-volume" pod "529cc133-aef7-4fa5-872b-c6e1713742be" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:13:54.898958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.898644 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:54.898958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.898743 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/529cc133-aef7-4fa5-872b-c6e1713742be-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:54.900032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.900009 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-kube-api-access-5vtn2" pod "529cc133-aef7-4fa5-872b-c6e1713742be" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be"). InnerVolumeSpecName "kube-api-access-5vtn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:54.900152 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:54.900133 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials" pod "529cc133-aef7-4fa5-872b-c6e1713742be" (UID: "529cc133-aef7-4fa5-872b-c6e1713742be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:13:55.000361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.000335 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:55.000361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.000361 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vtn2\" (UniqueName: \"kubernetes.io/projected/529cc133-aef7-4fa5-872b-c6e1713742be-kube-api-access-5vtn2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:13:55.061299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.061253 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:55.525862 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.525830 5090 generic.go:356] "Generic (PLEG): container finished" podID="529cc133-aef7-4fa5-872b-c6e1713742be" containerID="8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74" exitCode=0 Oct 31 02:13:55.526074 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.525878 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" Oct 31 02:13:55.526074 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.525917 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" event={"ID":"529cc133-aef7-4fa5-872b-c6e1713742be","Type":"ContainerDied","Data":"8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74"} Oct 31 02:13:55.526074 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.525975 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79" event={"ID":"529cc133-aef7-4fa5-872b-c6e1713742be","Type":"ContainerDied","Data":"1520b9b779923cb1fc628101d49c10882a765c6eff8672acc876afd9d454557d"} Oct 31 02:13:55.526074 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.526000 5090 scope.go:122] "RemoveContainer" containerID="8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74" Oct 31 02:13:55.537462 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.537445 5090 scope.go:122] "RemoveContainer" containerID="8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74" Oct 31 02:13:55.537781 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:13:55.537758 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74\": container with ID starting with 8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74 not found: ID does not exist" containerID="8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74" Oct 31 02:13:55.537870 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.537794 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74"} err="failed to get container status \"8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74\": rpc error: code = NotFound desc = could not find container \"8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74\": container with ID starting with 8900198e8009dfe0551e0c48d23736d8623262ce602c71a28d6f85a3cf785d74 not found: ID does not exist" Oct 31 02:13:55.549694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.549675 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79"] Oct 31 02:13:55.554080 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.554060 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0b6abf/payload-processing-external-model-e2e-extmod-b-0b6abf-5c48lcm79"] Oct 31 02:13:55.741601 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.741564 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:55.747451 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:55.747428 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529cc133-aef7-4fa5-872b-c6e1713742be" path="/var/lib/kubelet/pods/529cc133-aef7-4fa5-872b-c6e1713742be/volumes" Oct 31 02:13:56.352085 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:56.352040 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:57.024300 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:57.024267 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:57.658025 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:57.657984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:58.309854 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:58.309816 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:58.478748 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:58.477296 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:58.687321 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:58.687238 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:58.865789 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:58.865757 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:59.039468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:59.039434 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:13:59.533561 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:13:59.533515 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:00.354693 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:00.354648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:01.022760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:01.022678 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:01.493354 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:01.493324 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:14:01.547309 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:01.547208 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw"] Oct 31 02:14:01.747561 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:01.747485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:02.335223 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:02.335152 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:03.065612 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:03.065483 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:03.863414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:03.863375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:04.541308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:04.541125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:04.543339 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:04.543311 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:14:04.789434 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:14:04.789401 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:14:05.260688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:05.260652 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:05.881899 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:05.881866 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:06.018762 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.017188 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:06.157525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.157430 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:06.547962 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.547911 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" podUID="eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" containerName="payload-processing" containerID="cri-o://02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5" gracePeriod=25 Oct 31 02:14:06.858371 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.858309 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:06.878127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.878101 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:14:06.994869 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.994830 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-routing-overlay-volume\") pod \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " Oct 31 02:14:06.995346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.995016 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-plugins-config-volume\") pod \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " Oct 31 02:14:06.995346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.995149 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-routing-overlay-volume" pod "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" (UID: "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:06.995346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.995192 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-kube-api-access-sqjpx\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-kube-api-access-sqjpx\") pod \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " Oct 31 02:14:06.995346 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.995230 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-provider-credentials\") pod \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\" (UID: \"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3\") " Oct 31 02:14:06.995586 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.995346 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-plugins-config-volume" pod "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" (UID: "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:06.995730 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.995692 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:06.995803 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.995736 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:06.997668 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.997640 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-provider-credentials" pod "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" (UID: "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:06.997855 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:06.997668 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-kube-api-access-sqjpx" pod "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" (UID: "eddcaaed-cfe1-48d6-93be-ae92c74bf3b3"). InnerVolumeSpecName "kube-api-access-sqjpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:07.014047 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.014013 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:07.097549 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.097479 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqjpx\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-kube-api-access-sqjpx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:07.097549 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.097508 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:07.609504 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.609466 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:07.634220 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.634185 5090 generic.go:356] "Generic (PLEG): container finished" podID="eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" containerID="02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5" exitCode=0 Oct 31 02:14:07.634370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.634234 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" Oct 31 02:14:07.634370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.634270 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" event={"ID":"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3","Type":"ContainerDied","Data":"02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5"} Oct 31 02:14:07.634370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.634326 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw" event={"ID":"eddcaaed-cfe1-48d6-93be-ae92c74bf3b3","Type":"ContainerDied","Data":"4e3c17ea7cee3b80c78b700e93c0d30611d7568479c7f7094a07550a15280d76"} Oct 31 02:14:07.634370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.634356 5090 scope.go:122] "RemoveContainer" containerID="02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5" Oct 31 02:14:07.645830 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.645813 5090 scope.go:122] "RemoveContainer" containerID="02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5" Oct 31 02:14:07.646115 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:14:07.646095 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5\": container with ID starting with 02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5 not found: ID does not exist" containerID="02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5" Oct 31 02:14:07.646163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.646127 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5"} err="failed to get container status \"02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5\": rpc error: code = NotFound desc = could not find container \"02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5\": container with ID starting with 02eaaf6ab02dc79b623e1102258b3bddb840de3df1fdbdf941891e0fef4a0fb5 not found: ID does not exist" Oct 31 02:14:07.659852 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.659828 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw"] Oct 31 02:14:07.662994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.662973 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79f59b7c8-2qsvw"] Oct 31 02:14:07.748023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:07.747991 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" path="/var/lib/kubelet/pods/eddcaaed-cfe1-48d6-93be-ae92c74bf3b3/volumes" Oct 31 02:14:08.360562 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:08.360515 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:08.982075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:08.982031 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:09.690222 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:09.690180 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:10.186786 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.186746 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:10.223503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.223467 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6698479b-klcb5"] Oct 31 02:14:10.224836 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.224786 5090 cpu_manager.go:414] "containerMap: removing container" podUID="529cc133-aef7-4fa5-872b-c6e1713742be" containerName="payload-processing" Oct 31 02:14:10.225023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.225004 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529cc133-aef7-4fa5-872b-c6e1713742be" containerName="payload-processing" Oct 31 02:14:10.225134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.225121 5090 cpu_manager.go:414] "containerMap: removing container" podUID="eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" containerName="payload-processing" Oct 31 02:14:10.225228 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.225204 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" containerName="payload-processing" Oct 31 02:14:10.225385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.225365 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="529cc133-aef7-4fa5-872b-c6e1713742be" containerName="payload-processing" Oct 31 02:14:10.225454 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.225386 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="eddcaaed-cfe1-48d6-93be-ae92c74bf3b3" containerName="payload-processing" Oct 31 02:14:10.265084 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.261293 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6698479b-klcb5"] Oct 31 02:14:10.265084 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.261427 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.341744 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.341696 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:10.346278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.346252 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-plugins-config-volume\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.346654 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.346633 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-provider-credentials\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.346890 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.346872 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-routing-overlay-volume\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.347170 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.347150 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdxb7\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-kube-api-access-mdxb7\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.448768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.448670 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-plugins-config-volume\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.448768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.448766 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-provider-credentials\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.449026 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.448794 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-routing-overlay-volume\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.449026 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.448871 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdxb7\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-kube-api-access-mdxb7\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.449498 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.449471 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-routing-overlay-volume\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.449498 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.449507 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-plugins-config-volume\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.451755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.451695 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-provider-credentials\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.460867 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.460845 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdxb7\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-kube-api-access-mdxb7\") pod \"payload-processing-external-model-6c6698479b-klcb5\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.611381 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.611350 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:10.642486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.642450 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:10.940048 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:10.940018 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6698479b-klcb5"] Oct 31 02:14:10.941614 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:14:10.941585 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa20b91_0f02_481d_9e22_b5690eb78cf8.slice/crio-86bd58b08844f3a01573b4568ebabaf6fc87de679ab37b3e8ee4ce0603387951 WatchSource:0}: Error finding container 86bd58b08844f3a01573b4568ebabaf6fc87de679ab37b3e8ee4ce0603387951: Status 404 returned error can't find the container with id 86bd58b08844f3a01573b4568ebabaf6fc87de679ab37b3e8ee4ce0603387951 Oct 31 02:14:11.581412 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:11.581375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:11.712625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:11.712588 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" event={"ID":"4aa20b91-0f02-481d-9e22-b5690eb78cf8","Type":"ContainerStarted","Data":"a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c"} Oct 31 02:14:11.712810 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:11.712657 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" event={"ID":"4aa20b91-0f02-481d-9e22-b5690eb78cf8","Type":"ContainerStarted","Data":"86bd58b08844f3a01573b4568ebabaf6fc87de679ab37b3e8ee4ce0603387951"} Oct 31 02:14:11.712810 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:11.712687 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:11.734624 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:11.734576 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" podStartSLOduration=1.734563172 podStartE2EDuration="1.734563172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:14:11.731640187 +0000 UTC m=+3252.111885411" watchObservedRunningTime="2026-10-31 02:14:11.734563172 +0000 UTC m=+3252.114808417" Oct 31 02:14:11.958817 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:11.958736 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:12.100610 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:12.100575 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:12.271222 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:12.271194 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:12.412596 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:12.412557 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:12.588698 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:12.588085 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:12.952788 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:12.952365 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:13.133916 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.133881 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:13.283665 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.283626 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:13.851637 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.851602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:13.923320 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.923286 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj"] Oct 31 02:14:13.927762 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.927742 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:13.931143 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.931078 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c088fd\"/\"agc-cred-16730c8873e2\"" Oct 31 02:14:13.933854 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.933827 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c088fd\"/\"payload-processing-external-model-e2e--e07e76ef-dockercfg-h2mt7\"" Oct 31 02:14:13.937010 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.936980 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c088fd\"/\"routing-overlay\"" Oct 31 02:14:13.937319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.937253 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c088fd\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c088fd\"" Oct 31 02:14:13.937319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.937269 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c088fd\"/\"openshift-service-ca.crt\"" Oct 31 02:14:13.937514 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.937479 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c088fd\"/\"kube-root-ca.crt\"" Oct 31 02:14:13.941795 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:13.941744 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj"] Oct 31 02:14:14.045307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.045271 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.045470 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.045326 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdth5\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-kube-api-access-mdth5\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.045470 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.045460 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.045553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.045492 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.057081 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.057048 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:14.165073 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.164963 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.165671 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.165628 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.170774 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.166235 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.192842 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.182605 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.213831 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.213785 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.214023 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.213887 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdth5\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-kube-api-access-mdth5\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.215594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.215562 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.228304 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.228277 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdth5\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-kube-api-access-mdth5\") pod \"payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.241398 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.241375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:14.244784 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.244762 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.306626 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.306577 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl"] Oct 31 02:14:14.312965 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.312766 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.330103 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.329903 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl"] Oct 31 02:14:14.330103 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.330080 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"kube-root-ca.crt\"" Oct 31 02:14:14.333769 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.330519 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"routing-overlay\"" Oct 31 02:14:14.333769 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.330732 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-28e4f3\"" Oct 31 02:14:14.333769 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.330748 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"openshift-service-ca.crt\"" Oct 31 02:14:14.356743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.353952 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-28e4f3\"/\"agc-cred-12d790434958\"" Oct 31 02:14:14.418207 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.418121 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.418338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.418279 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.418338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.418310 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.418488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.418470 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tg29\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-kube-api-access-4tg29\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.524741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.524562 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.524741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.524622 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.525026 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.524770 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tg29\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-kube-api-access-4tg29\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.525026 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.524922 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.531768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.528330 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.531768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.529297 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.560661 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.560628 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tg29\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-kube-api-access-4tg29\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.565518 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.565456 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.607676 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.607644 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj"] Oct 31 02:14:14.610461 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:14:14.610415 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca05e63_f923_4eec_8c2e_f52416ba67f4.slice/crio-11397ea8d3606b1e8dc0fb261896dec03dbcd00901916414adcd139ca16c062b WatchSource:0}: Error finding container 11397ea8d3606b1e8dc0fb261896dec03dbcd00901916414adcd139ca16c062b: Status 404 returned error can't find the container with id 11397ea8d3606b1e8dc0fb261896dec03dbcd00901916414adcd139ca16c062b Oct 31 02:14:14.683417 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.683391 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:14.737784 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.735937 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" event={"ID":"3ca05e63-f923-4eec-8c2e-f52416ba67f4","Type":"ContainerStarted","Data":"acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09"} Oct 31 02:14:14.737784 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.735991 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" event={"ID":"3ca05e63-f923-4eec-8c2e-f52416ba67f4","Type":"ContainerStarted","Data":"11397ea8d3606b1e8dc0fb261896dec03dbcd00901916414adcd139ca16c062b"} Oct 31 02:14:14.737784 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.736025 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:14.766277 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.766226 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" podStartSLOduration=1.766210585 podStartE2EDuration="1.766210585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:14:14.75924841 +0000 UTC m=+3255.139493650" watchObservedRunningTime="2026-10-31 02:14:14.766210585 +0000 UTC m=+3255.146455803" Oct 31 02:14:14.854102 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.854072 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:14.970843 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:14.970805 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl"] Oct 31 02:14:14.972221 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:14:14.972190 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a83099b_12c5_403d_8695_923fdce5db09.slice/crio-29bff9a09701a2999a90f31db784952209c22c5337215e38fa967ed0054e5cfe WatchSource:0}: Error finding container 29bff9a09701a2999a90f31db784952209c22c5337215e38fa967ed0054e5cfe: Status 404 returned error can't find the container with id 29bff9a09701a2999a90f31db784952209c22c5337215e38fa967ed0054e5cfe Oct 31 02:14:15.570767 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:15.570675 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:15.747213 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:15.747180 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" event={"ID":"8a83099b-12c5-403d-8695-923fdce5db09","Type":"ContainerStarted","Data":"6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0"} Oct 31 02:14:15.747392 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:15.747229 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" event={"ID":"8a83099b-12c5-403d-8695-923fdce5db09","Type":"ContainerStarted","Data":"29bff9a09701a2999a90f31db784952209c22c5337215e38fa967ed0054e5cfe"} Oct 31 02:14:15.765278 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:15.765230 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" podStartSLOduration=1.765218116 podStartE2EDuration="1.765218116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:14:15.762956363 +0000 UTC m=+3256.143201587" watchObservedRunningTime="2026-10-31 02:14:15.765218116 +0000 UTC m=+3256.145463338" Oct 31 02:14:16.390226 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:16.390187 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:16.529881 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:16.529845 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:16.756648 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:16.756618 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:17.072194 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:17.072092 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:17.758963 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:17.755365 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:18.353467 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:18.348016 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:19.045611 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.045582 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:19.519062 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.519025 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs"] Oct 31 02:14:19.519305 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.519273 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" podUID="8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" containerName="maas-api" containerID="cri-o://e5ce57c17c029fb8465f2f507e45caa1583ed90d6a58af7947b054b24845b74f" gracePeriod=30 Oct 31 02:14:19.593532 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.593496 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:19.861800 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.860594 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:19.897240 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.897199 5090 generic.go:356] "Generic (PLEG): container finished" podID="8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" containerID="e5ce57c17c029fb8465f2f507e45caa1583ed90d6a58af7947b054b24845b74f" exitCode=0 Oct 31 02:14:19.897634 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.897613 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" event={"ID":"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e","Type":"ContainerDied","Data":"e5ce57c17c029fb8465f2f507e45caa1583ed90d6a58af7947b054b24845b74f"} Oct 31 02:14:19.996760 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:19.996733 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 02:14:20.009900 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.009871 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:20.086144 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.086111 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-kube-api-access-xjhhx\" (UniqueName: \"kubernetes.io/projected/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-kube-api-access-xjhhx\") pod \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " Oct 31 02:14:20.086623 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.086180 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-metrics-certs\") pod \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " Oct 31 02:14:20.086623 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.086381 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-maas-api-tls\") pod \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " Oct 31 02:14:20.086623 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.086467 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-openshift-service-ca\") pod \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\" (UID: \"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e\") " Oct 31 02:14:20.087117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.087077 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-openshift-service-ca" pod "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" (UID: "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:20.088810 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.088783 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-maas-api-tls" pod "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" (UID: "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:20.089246 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.089222 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-metrics-certs" pod "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" (UID: "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:20.089333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.089276 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-kube-api-access-xjhhx" pod "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" (UID: "8fb91b77-ea0e-4a2d-9402-41e876ff6f9e"). InnerVolumeSpecName "kube-api-access-xjhhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:20.188390 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.188296 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhhx\" (UniqueName: \"kubernetes.io/projected/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-kube-api-access-xjhhx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:20.188390 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.188333 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:20.188390 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.188343 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:20.188390 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.188352 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:20.201603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.201560 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:20.459178 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.459091 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:20.616900 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.616869 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:20.773681 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.773641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:20.912427 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.912347 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:20.925876 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.925835 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" Oct 31 02:14:20.926092 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.925053 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs" event={"ID":"8fb91b77-ea0e-4a2d-9402-41e876ff6f9e","Type":"ContainerDied","Data":"031e6a9c6c2eefd3bc787ffc0168364f6a3d5aabd5c14e66935d7cae7769553c"} Oct 31 02:14:20.926172 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.926120 5090 scope.go:122] "RemoveContainer" containerID="e5ce57c17c029fb8465f2f507e45caa1583ed90d6a58af7947b054b24845b74f" Oct 31 02:14:20.959308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.959277 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs"] Oct 31 02:14:20.964092 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:20.964061 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-be6935-5494f85745-rc7rs"] Oct 31 02:14:21.066826 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:21.066707 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:21.260636 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:21.260602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:21.748788 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:21.748758 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" path="/var/lib/kubelet/pods/8fb91b77-ea0e-4a2d-9402-41e876ff6f9e/volumes" Oct 31 02:14:21.833028 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:21.832997 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:22.684967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:22.684933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:22.723033 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:22.723004 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:14:22.774761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:22.774709 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f"] Oct 31 02:14:22.825691 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:22.825654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:23.472988 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:23.472956 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:24.155142 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:24.155107 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:24.873745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:24.870679 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:25.589684 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:25.589649 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:25.748686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:25.748652 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:14:26.449069 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:26.449036 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:27.156672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:27.156636 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:27.775257 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:27.775174 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" podUID="e6b85e89-5c51-46ac-b7e8-7128e4d773bf" containerName="payload-processing" containerID="cri-o://2d301d7830b1901a175207219116c196ceead62dd3c70ec69d8974aa5e2824a5" gracePeriod=25 Oct 31 02:14:27.781207 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:27.781179 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:14:27.998236 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:27.998075 5090 generic.go:356] "Generic (PLEG): container finished" podID="e6b85e89-5c51-46ac-b7e8-7128e4d773bf" containerID="2d301d7830b1901a175207219116c196ceead62dd3c70ec69d8974aa5e2824a5" exitCode=0 Oct 31 02:14:27.998236 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:27.998161 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" event={"ID":"e6b85e89-5c51-46ac-b7e8-7128e4d773bf","Type":"ContainerDied","Data":"2d301d7830b1901a175207219116c196ceead62dd3c70ec69d8974aa5e2824a5"} Oct 31 02:14:28.032302 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.031803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:28.130915 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.130887 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:14:28.219362 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.219327 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:28.260550 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.260498 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-provider-credentials\") pod \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " Oct 31 02:14:28.260809 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.260779 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-routing-overlay-volume\") pod \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " Oct 31 02:14:28.260903 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.260868 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-kube-api-access-gl6dv\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-kube-api-access-gl6dv\") pod \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " Oct 31 02:14:28.260966 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.260936 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-plugins-config-volume\") pod \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\" (UID: \"e6b85e89-5c51-46ac-b7e8-7128e4d773bf\") " Oct 31 02:14:28.261757 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.261705 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-routing-overlay-volume" pod "e6b85e89-5c51-46ac-b7e8-7128e4d773bf" (UID: "e6b85e89-5c51-46ac-b7e8-7128e4d773bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:28.262847 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.262817 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:28.263126 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.263105 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-plugins-config-volume" pod "e6b85e89-5c51-46ac-b7e8-7128e4d773bf" (UID: "e6b85e89-5c51-46ac-b7e8-7128e4d773bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:28.269789 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.265502 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-kube-api-access-gl6dv" pod "e6b85e89-5c51-46ac-b7e8-7128e4d773bf" (UID: "e6b85e89-5c51-46ac-b7e8-7128e4d773bf"). InnerVolumeSpecName "kube-api-access-gl6dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:28.343814 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.342972 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-provider-credentials" pod "e6b85e89-5c51-46ac-b7e8-7128e4d773bf" (UID: "e6b85e89-5c51-46ac-b7e8-7128e4d773bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:28.381583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.372406 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl6dv\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-kube-api-access-gl6dv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:28.381583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.372450 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:28.381583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.372469 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e6b85e89-5c51-46ac-b7e8-7128e4d773bf-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:28.381583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.372688 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:28.387845 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.387813 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2"] Oct 31 02:14:28.388582 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.388498 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" containerID="cri-o://2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b" gracePeriod=30 Oct 31 02:14:28.528361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:28.528324 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:29.045741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:29.045466 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" event={"ID":"e6b85e89-5c51-46ac-b7e8-7128e4d773bf","Type":"ContainerDied","Data":"39bc2212e0da7967b0b33a675dc344e06e9cee5ca22f7459f5b1b8c373ebc236"} Oct 31 02:14:29.045741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:29.045543 5090 scope.go:122] "RemoveContainer" containerID="2d301d7830b1901a175207219116c196ceead62dd3c70ec69d8974aa5e2824a5" Oct 31 02:14:29.046028 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:29.045758 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f" Oct 31 02:14:29.130754 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:29.124163 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f"] Oct 31 02:14:29.134369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:29.134305 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fd8fb4945-wsw8f"] Oct 31 02:14:29.300795 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:29.293504 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:29.761743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:29.761690 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b85e89-5c51-46ac-b7e8-7128e4d773bf" path="/var/lib/kubelet/pods/e6b85e89-5c51-46ac-b7e8-7128e4d773bf/volumes" Oct 31 02:14:30.426786 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:30.426538 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:30.563510 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:30.563477 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:31.112882 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.112847 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:31.256453 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.256424 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:31.548386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.548351 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm"] Oct 31 02:14:31.548946 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.548647 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" podUID="141bfd4e-2278-46a6-ba07-fb9651d5766e" containerName="maas-api" containerID="cri-o://940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965" gracePeriod=30 Oct 31 02:14:31.707550 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.707519 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:31.895738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.887431 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 02:14:31.929739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.925331 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.985525 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-maas-api-tls\") pod \"141bfd4e-2278-46a6-ba07-fb9651d5766e\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.985638 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141bfd4e-2278-46a6-ba07-fb9651d5766e-kube-api-access-74m8n\" (UniqueName: \"kubernetes.io/projected/141bfd4e-2278-46a6-ba07-fb9651d5766e-kube-api-access-74m8n\") pod \"141bfd4e-2278-46a6-ba07-fb9651d5766e\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.985822 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141bfd4e-2278-46a6-ba07-fb9651d5766e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bfd4e-2278-46a6-ba07-fb9651d5766e-openshift-service-ca\") pod \"141bfd4e-2278-46a6-ba07-fb9651d5766e\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.986072 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-metrics-certs\") pod \"141bfd4e-2278-46a6-ba07-fb9651d5766e\" (UID: \"141bfd4e-2278-46a6-ba07-fb9651d5766e\") " Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.989499 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-metrics-certs" pod "141bfd4e-2278-46a6-ba07-fb9651d5766e" (UID: "141bfd4e-2278-46a6-ba07-fb9651d5766e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.992080 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-maas-api-tls" pod "141bfd4e-2278-46a6-ba07-fb9651d5766e" (UID: "141bfd4e-2278-46a6-ba07-fb9651d5766e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.992574 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141bfd4e-2278-46a6-ba07-fb9651d5766e-openshift-service-ca" pod "141bfd4e-2278-46a6-ba07-fb9651d5766e" (UID: "141bfd4e-2278-46a6-ba07-fb9651d5766e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:31.998494 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:31.995251 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141bfd4e-2278-46a6-ba07-fb9651d5766e-kube-api-access-74m8n" pod "141bfd4e-2278-46a6-ba07-fb9651d5766e" (UID: "141bfd4e-2278-46a6-ba07-fb9651d5766e"). InnerVolumeSpecName "kube-api-access-74m8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:32.081556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.081510 5090 generic.go:356] "Generic (PLEG): container finished" podID="141bfd4e-2278-46a6-ba07-fb9651d5766e" containerID="940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965" exitCode=0 Oct 31 02:14:32.081801 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.081663 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" event={"ID":"141bfd4e-2278-46a6-ba07-fb9651d5766e","Type":"ContainerDied","Data":"940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965"} Oct 31 02:14:32.081801 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.081702 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" event={"ID":"141bfd4e-2278-46a6-ba07-fb9651d5766e","Type":"ContainerDied","Data":"8c02aedc363edac44147b53dbbf39dc4730b73eae0e4dfb80864cdaf07e43c86"} Oct 31 02:14:32.081801 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.081753 5090 scope.go:122] "RemoveContainer" containerID="940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965" Oct 31 02:14:32.082011 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.081841 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm" Oct 31 02:14:32.089079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.088147 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:32.089079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.089013 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/141bfd4e-2278-46a6-ba07-fb9651d5766e-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:32.089079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.089035 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74m8n\" (UniqueName: \"kubernetes.io/projected/141bfd4e-2278-46a6-ba07-fb9651d5766e-kube-api-access-74m8n\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:32.089079 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.089053 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/141bfd4e-2278-46a6-ba07-fb9651d5766e-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:32.095673 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.095652 5090 scope.go:122] "RemoveContainer" containerID="940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965" Oct 31 02:14:32.095959 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:14:32.095931 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965\": container with ID starting with 940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965 not found: ID does not exist" containerID="940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965" Oct 31 02:14:32.096068 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.095968 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965"} err="failed to get container status \"940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965\": rpc error: code = NotFound desc = could not find container \"940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965\": container with ID starting with 940077072ac9e3e1c9c54745b2305399ee42fbaf2fc89b9f8de5b1e88b841965 not found: ID does not exist" Oct 31 02:14:32.109686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.109659 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm"] Oct 31 02:14:32.115203 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.115181 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ac7e0d-6c7c464f44-57kdm"] Oct 31 02:14:32.130599 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.130574 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:32.271638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.271600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:32.671169 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.671137 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:32.843886 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:32.841351 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:33.633044 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.631789 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:33.716440 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.716411 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 02:14:33.749392 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.749360 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141bfd4e-2278-46a6-ba07-fb9651d5766e" path="/var/lib/kubelet/pods/141bfd4e-2278-46a6-ba07-fb9651d5766e/volumes" Oct 31 02:14:33.833575 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833546 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-data\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.833758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833588 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-socket\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.833758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833611 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-credential-socket\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.833758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833645 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-kube-api-access-9wj9f\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-kube-api-access-9wj9f\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.833758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833677 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb292c29-cded-40f7-b40a-086df28415db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb292c29-cded-40f7-b40a-086df28415db-istio-podinfo\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.834015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833803 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-istio-token\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-istio-token\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.834015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833892 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-data" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:33.834015 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833911 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-envoy\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.834174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833934 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-socket" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:33.834174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833986 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-credential-socket" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:33.834174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.833961 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb292c29-cded-40f7-b40a-086df28415db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb292c29-cded-40f7-b40a-086df28415db-istiod-ca-cert\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.834174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.834134 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-certs\") pod \"bb292c29-cded-40f7-b40a-086df28415db\" (UID: \"bb292c29-cded-40f7-b40a-086df28415db\") " Oct 31 02:14:33.834359 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.834297 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb292c29-cded-40f7-b40a-086df28415db-istiod-ca-cert" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:33.834922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.834603 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-certs" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:33.835078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.835051 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb292c29-cded-40f7-b40a-086df28415db-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.835165 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.835084 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.835165 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.835100 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.835165 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.835116 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.835165 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.835129 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.836900 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.836866 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-envoy" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:14:33.837003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.836933 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-istio-token" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:33.837076 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.837053 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-kube-api-access-9wj9f" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "kube-api-access-9wj9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:33.837313 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.837290 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb292c29-cded-40f7-b40a-086df28415db-istio-podinfo" pod "bb292c29-cded-40f7-b40a-086df28415db" (UID: "bb292c29-cded-40f7-b40a-086df28415db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:14:33.936198 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.936161 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wj9f\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-kube-api-access-9wj9f\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.936198 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.936195 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb292c29-cded-40f7-b40a-086df28415db-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.936361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.936210 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb292c29-cded-40f7-b40a-086df28415db-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:33.936361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:33.936223 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb292c29-cded-40f7-b40a-086df28415db-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:34.096834 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.096800 5090 generic.go:356] "Generic (PLEG): container finished" podID="bb292c29-cded-40f7-b40a-086df28415db" containerID="2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b" exitCode=0 Oct 31 02:14:34.097005 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.096855 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" Oct 31 02:14:34.097005 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.096897 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" event={"ID":"bb292c29-cded-40f7-b40a-086df28415db","Type":"ContainerDied","Data":"2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b"} Oct 31 02:14:34.097005 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.096960 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2" event={"ID":"bb292c29-cded-40f7-b40a-086df28415db","Type":"ContainerDied","Data":"38e80d8b7b265589464f642ff7fe77ca6fe94081016c0d19d3c89a91932dbdd8"} Oct 31 02:14:34.097005 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.096991 5090 scope.go:122] "RemoveContainer" containerID="2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b" Oct 31 02:14:34.110093 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.110075 5090 scope.go:122] "RemoveContainer" containerID="2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b" Oct 31 02:14:34.110401 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:14:34.110378 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b\": container with ID starting with 2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b not found: ID does not exist" containerID="2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b" Oct 31 02:14:34.110472 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.110410 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b"} err="failed to get container status \"2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b\": rpc error: code = NotFound desc = could not find container \"2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b\": container with ID starting with 2301e744877bc7797c734f94c7e4bb7a06df0efb800f6ef58254edf6f948f72b not found: ID does not exist" Oct 31 02:14:34.126055 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.126021 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2"] Oct 31 02:14:34.140150 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.140113 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be6935-openshift-default-7bff7f4c57-x7tz2"] Oct 31 02:14:34.377882 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:34.377783 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:34.790031 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:14:34.789996 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:14:35.139688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:35.138585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:35.751365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:35.751334 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb292c29-cded-40f7-b40a-086df28415db" path="/var/lib/kubelet/pods/bb292c29-cded-40f7-b40a-086df28415db/volumes" Oct 31 02:14:35.771356 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:35.771327 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:36.472752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:36.467109 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:37.347763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:37.344418 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:37.948007 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:37.947947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:38.726280 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:38.726200 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:39.380505 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:39.380462 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:39.523347 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:39.523303 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:39.767362 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:39.767332 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:40.247947 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:40.247910 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:41.026796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:41.026761 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:41.674901 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:41.674274 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:42.336738 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:42.336687 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:43.056259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:43.056222 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:43.798525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:43.798301 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:44.575741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:44.574249 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:45.255754 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:45.255693 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:45.858509 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:45.858468 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:46.456904 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:46.456854 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:47.157651 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:47.157619 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:47.800309 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:47.800255 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:48.438556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:48.438520 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:49.142123 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:49.142071 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:49.963853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:49.963676 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:50.498197 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:50.498163 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:50.636705 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:50.636677 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:50.785813 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:50.784963 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:51.187789 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.186948 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv"] Oct 31 02:14:51.187789 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.187198 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" podUID="fec35109-1f78-4101-a506-66372eb75e36" containerName="maas-api" containerID="cri-o://45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b" gracePeriod=30 Oct 31 02:14:51.256497 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.256464 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:51.413690 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.413592 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:51.574383 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.574347 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:51.614654 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.614632 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 02:14:51.688771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.686778 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-maas-api-tls\") pod \"fec35109-1f78-4101-a506-66372eb75e36\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " Oct 31 02:14:51.688771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.686897 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-metrics-certs\") pod \"fec35109-1f78-4101-a506-66372eb75e36\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " Oct 31 02:14:51.688771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.686920 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec35109-1f78-4101-a506-66372eb75e36-kube-api-access-bfv2v\" (UniqueName: \"kubernetes.io/projected/fec35109-1f78-4101-a506-66372eb75e36-kube-api-access-bfv2v\") pod \"fec35109-1f78-4101-a506-66372eb75e36\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " Oct 31 02:14:51.688771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.687019 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec35109-1f78-4101-a506-66372eb75e36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec35109-1f78-4101-a506-66372eb75e36-openshift-service-ca\") pod \"fec35109-1f78-4101-a506-66372eb75e36\" (UID: \"fec35109-1f78-4101-a506-66372eb75e36\") " Oct 31 02:14:51.688771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.687680 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec35109-1f78-4101-a506-66372eb75e36-openshift-service-ca" pod "fec35109-1f78-4101-a506-66372eb75e36" (UID: "fec35109-1f78-4101-a506-66372eb75e36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:14:51.691542 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.691510 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec35109-1f78-4101-a506-66372eb75e36-kube-api-access-bfv2v" pod "fec35109-1f78-4101-a506-66372eb75e36" (UID: "fec35109-1f78-4101-a506-66372eb75e36"). InnerVolumeSpecName "kube-api-access-bfv2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:14:51.691700 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.691647 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-metrics-certs" pod "fec35109-1f78-4101-a506-66372eb75e36" (UID: "fec35109-1f78-4101-a506-66372eb75e36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:51.692675 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.692645 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-maas-api-tls" pod "fec35109-1f78-4101-a506-66372eb75e36" (UID: "fec35109-1f78-4101-a506-66372eb75e36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:14:51.718233 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.718202 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:51.788214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.788181 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec35109-1f78-4101-a506-66372eb75e36-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:51.788417 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.788318 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:51.788417 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.788350 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fec35109-1f78-4101-a506-66372eb75e36-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:51.788417 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:51.788368 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfv2v\" (UniqueName: \"kubernetes.io/projected/fec35109-1f78-4101-a506-66372eb75e36-kube-api-access-bfv2v\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:14:52.111615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.111525 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:52.246010 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.245977 5090 generic.go:356] "Generic (PLEG): container finished" podID="fec35109-1f78-4101-a506-66372eb75e36" containerID="45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b" exitCode=0 Oct 31 02:14:52.246227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.246068 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" Oct 31 02:14:52.246227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.246070 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" event={"ID":"fec35109-1f78-4101-a506-66372eb75e36","Type":"ContainerDied","Data":"45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b"} Oct 31 02:14:52.246227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.246183 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" event={"ID":"fec35109-1f78-4101-a506-66372eb75e36","Type":"ContainerDied","Data":"46ac9cb7137db706318c6193e4e5a631a0dba0acfd45ea2d4b6ee4d6e3401758"} Oct 31 02:14:52.246227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.246226 5090 scope.go:122] "RemoveContainer" containerID="45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b" Oct 31 02:14:52.256564 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.256546 5090 scope.go:122] "RemoveContainer" containerID="45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b" Oct 31 02:14:52.256736 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.256703 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:52.256929 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:14:52.256911 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b\": container with ID starting with 45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b not found: ID does not exist" containerID="45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b" Oct 31 02:14:52.257030 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.256937 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b"} err="failed to get container status \"45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b\": rpc error: code = NotFound desc = could not find container \"45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b\": container with ID starting with 45339fbb5c3d7a86390c37a0be542bd6bad07dd4425ade9805ae50e55fc5282b not found: ID does not exist" Oct 31 02:14:52.265518 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.265495 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv"] Oct 31 02:14:52.271627 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.271596 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv"] Oct 31 02:14:52.395402 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:52.395314 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:53.031919 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:53.031880 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:53.664363 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:53.664329 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:53.748750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:53.748689 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec35109-1f78-4101-a506-66372eb75e36" path="/var/lib/kubelet/pods/fec35109-1f78-4101-a506-66372eb75e36/volumes" Oct 31 02:14:54.352873 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:54.352841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:54.498013 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:54.497968 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-0b6abf-8bf7869bd-589bv" podUID="fec35109-1f78-4101-a506-66372eb75e36" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.151:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 02:14:55.017133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:55.014658 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:55.625469 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:55.625424 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:56.284139 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:56.284103 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:56.954558 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:56.954464 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:57.651745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:57.651681 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:58.371898 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:58.371850 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:59.123025 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:59.122964 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:14:59.863078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:14:59.863036 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:00.005145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:00.005114 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:00.378454 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:00.378411 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:00.503658 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:00.503625 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:15:00.512250 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:00.512225 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:15:00.531750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:00.531693 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:15:00.536759 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:00.536707 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:15:00.769732 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:00.765339 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:01.642596 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:01.642499 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:02.439567 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:02.439527 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:03.069860 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:03.069828 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:03.751427 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:03.751399 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:04.753421 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:04.753383 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:04.793141 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:15:04.793099 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:15:05.576502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:05.576439 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:06.404615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:06.404577 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:07.049699 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:07.049657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:07.690594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:07.690555 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:08.362677 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:08.361901 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:09.065677 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:09.065631 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:09.734084 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:09.734048 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:10.348009 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:10.347974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:10.971413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:10.971375 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:11.189687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:11.189649 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:11.346822 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:11.339208 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:11.758290 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:11.758238 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:12.516455 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:12.516419 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:13.218581 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:13.218541 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:13.827922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:13.827886 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:14.455381 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:14.455347 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:15.152110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:15.152056 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:16.035519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:16.035481 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:16.765825 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:16.765783 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:17.649740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:17.646681 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:18.344010 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:18.343977 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:19.022909 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:19.022873 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:19.484670 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:19.484585 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:19.625639 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:19.625607 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:20.015379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.015343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:20.237401 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.237341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:20.295460 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.295197 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4"] Oct 31 02:15:20.296593 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.296545 5090 cpu_manager.go:414] "containerMap: removing container" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" Oct 31 02:15:20.296826 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.296807 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" Oct 31 02:15:20.296962 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.296950 5090 cpu_manager.go:414] "containerMap: removing container" podUID="141bfd4e-2278-46a6-ba07-fb9651d5766e" containerName="maas-api" Oct 31 02:15:20.297056 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297044 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141bfd4e-2278-46a6-ba07-fb9651d5766e" containerName="maas-api" Oct 31 02:15:20.297167 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297156 5090 cpu_manager.go:414] "containerMap: removing container" podUID="e6b85e89-5c51-46ac-b7e8-7128e4d773bf" containerName="payload-processing" Oct 31 02:15:20.297263 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297249 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b85e89-5c51-46ac-b7e8-7128e4d773bf" containerName="payload-processing" Oct 31 02:15:20.297365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297354 5090 cpu_manager.go:414] "containerMap: removing container" podUID="fec35109-1f78-4101-a506-66372eb75e36" containerName="maas-api" Oct 31 02:15:20.297448 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297437 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec35109-1f78-4101-a506-66372eb75e36" containerName="maas-api" Oct 31 02:15:20.297538 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297527 5090 cpu_manager.go:414] "containerMap: removing container" podUID="8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" containerName="maas-api" Oct 31 02:15:20.297619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297609 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" containerName="maas-api" Oct 31 02:15:20.297894 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.297877 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="141bfd4e-2278-46a6-ba07-fb9651d5766e" containerName="maas-api" Oct 31 02:15:20.298097 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.298082 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b85e89-5c51-46ac-b7e8-7128e4d773bf" containerName="payload-processing" Oct 31 02:15:20.298202 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.298190 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec35109-1f78-4101-a506-66372eb75e36" containerName="maas-api" Oct 31 02:15:20.298279 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.298271 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb91b77-ea0e-4a2d-9402-41e876ff6f9e" containerName="maas-api" Oct 31 02:15:20.298353 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.298345 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb292c29-cded-40f7-b40a-086df28415db" containerName="istio-proxy" Oct 31 02:15:20.332100 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.332068 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4"] Oct 31 02:15:20.332474 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.332452 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.346037 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.346000 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-91b6b1\"" Oct 31 02:15:20.346287 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.346261 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"routing-overlay\"" Oct 31 02:15:20.346389 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.346368 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"kube-root-ca.crt\"" Oct 31 02:15:20.346456 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.346437 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"openshift-service-ca.crt\"" Oct 31 02:15:20.346456 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.346131 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"agc-cred-686598284250\"" Oct 31 02:15:20.346643 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.346624 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91b6b1\"/\"payload-processing-external-model-e2e--e5c46109-dockercfg-8pnsm\"" Oct 31 02:15:20.403289 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.403252 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.403488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.403360 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.403488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.403473 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.403635 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.403566 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dtkb\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-kube-api-access-8dtkb\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.504986 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.504945 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.505499 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.505458 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.505660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.505515 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dtkb\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-kube-api-access-8dtkb\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.505988 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.505962 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.506786 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.506759 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.506983 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.506953 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.508284 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.508258 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.521302 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.521271 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dtkb\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-kube-api-access-8dtkb\") pod \"payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.689054 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.688962 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:20.954581 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.954429 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:20.995039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:20.995009 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4"] Oct 31 02:15:21.477993 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:21.477941 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" event={"ID":"61dc5b10-f1f1-4eea-897a-b51407a7ef17","Type":"ContainerStarted","Data":"a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a"} Oct 31 02:15:21.478517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:21.478016 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" event={"ID":"61dc5b10-f1f1-4eea-897a-b51407a7ef17","Type":"ContainerStarted","Data":"241a72d9e7f018264e3f9715b84f3b82692194868b65b701f9fe9d076737c2ba"} Oct 31 02:15:21.478517 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:21.478078 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:21.499136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:21.499071 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" podStartSLOduration=1.499053291 podStartE2EDuration="1.499053291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:15:21.496947653 +0000 UTC m=+3321.877192905" watchObservedRunningTime="2026-10-31 02:15:21.499053291 +0000 UTC m=+3321.879298515" Oct 31 02:15:21.771125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:21.771026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:22.539025 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:22.538959 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:23.229133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:23.229099 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:23.969298 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:23.969227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:24.641665 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:24.641617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:25.196779 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.195303 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg"] Oct 31 02:15:25.196779 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.195618 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" podUID="d46cdd57-3bb7-4b44-863c-73c12433a40d" containerName="maas-api" containerID="cri-o://9ae6dd245597836a1bf18a552bd9291a9172d3e64802797595174c1f95cadb91" gracePeriod=30 Oct 31 02:15:25.272369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.272331 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:25.429221 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.421020 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:25.558692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.558419 5090 generic.go:356] "Generic (PLEG): container finished" podID="d46cdd57-3bb7-4b44-863c-73c12433a40d" containerID="9ae6dd245597836a1bf18a552bd9291a9172d3e64802797595174c1f95cadb91" exitCode=0 Oct 31 02:15:25.558692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.558610 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" event={"ID":"d46cdd57-3bb7-4b44-863c-73c12433a40d","Type":"ContainerDied","Data":"9ae6dd245597836a1bf18a552bd9291a9172d3e64802797595174c1f95cadb91"} Oct 31 02:15:25.583619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.583513 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:25.670639 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.670584 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 02:15:25.723808 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.723697 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:25.811076 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.810996 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46cdd57-3bb7-4b44-863c-73c12433a40d-kube-api-access-bwjr7\" (UniqueName: \"kubernetes.io/projected/d46cdd57-3bb7-4b44-863c-73c12433a40d-kube-api-access-bwjr7\") pod \"d46cdd57-3bb7-4b44-863c-73c12433a40d\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " Oct 31 02:15:25.811076 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.811055 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-metrics-certs\") pod \"d46cdd57-3bb7-4b44-863c-73c12433a40d\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " Oct 31 02:15:25.811338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.811126 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-maas-api-tls\") pod \"d46cdd57-3bb7-4b44-863c-73c12433a40d\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " Oct 31 02:15:25.811338 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.811152 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46cdd57-3bb7-4b44-863c-73c12433a40d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46cdd57-3bb7-4b44-863c-73c12433a40d-openshift-service-ca\") pod \"d46cdd57-3bb7-4b44-863c-73c12433a40d\" (UID: \"d46cdd57-3bb7-4b44-863c-73c12433a40d\") " Oct 31 02:15:25.811623 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.811592 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46cdd57-3bb7-4b44-863c-73c12433a40d-openshift-service-ca" pod "d46cdd57-3bb7-4b44-863c-73c12433a40d" (UID: "d46cdd57-3bb7-4b44-863c-73c12433a40d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:15:25.811917 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.811898 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d46cdd57-3bb7-4b44-863c-73c12433a40d-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:25.813468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.813440 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46cdd57-3bb7-4b44-863c-73c12433a40d-kube-api-access-bwjr7" pod "d46cdd57-3bb7-4b44-863c-73c12433a40d" (UID: "d46cdd57-3bb7-4b44-863c-73c12433a40d"). InnerVolumeSpecName "kube-api-access-bwjr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:15:25.817134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.817113 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-metrics-certs" pod "d46cdd57-3bb7-4b44-863c-73c12433a40d" (UID: "d46cdd57-3bb7-4b44-863c-73c12433a40d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:15:25.817242 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.817139 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-maas-api-tls" pod "d46cdd57-3bb7-4b44-863c-73c12433a40d" (UID: "d46cdd57-3bb7-4b44-863c-73c12433a40d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:15:25.913340 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.913303 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwjr7\" (UniqueName: \"kubernetes.io/projected/d46cdd57-3bb7-4b44-863c-73c12433a40d-kube-api-access-bwjr7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:25.913340 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.913342 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:25.913582 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:25.913358 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d46cdd57-3bb7-4b44-863c-73c12433a40d-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:26.096440 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:26.096361 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:26.574969 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:26.574928 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:26.578529 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:26.578498 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" event={"ID":"d46cdd57-3bb7-4b44-863c-73c12433a40d","Type":"ContainerDied","Data":"43a3237b8eccf608d8d4ed52aab45c4648372a7a71ab83a8591ec4e5fd7d6d44"} Oct 31 02:15:26.578529 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:26.578522 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg" Oct 31 02:15:26.578818 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:26.578565 5090 scope.go:122] "RemoveContainer" containerID="9ae6dd245597836a1bf18a552bd9291a9172d3e64802797595174c1f95cadb91" Oct 31 02:15:26.605283 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:26.605261 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg"] Oct 31 02:15:26.611435 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:26.611403 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-6d7f5d-64485f848b-c5sbg"] Oct 31 02:15:27.246749 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:27.244825 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:27.746680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:27.746645 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46cdd57-3bb7-4b44-863c-73c12433a40d" path="/var/lib/kubelet/pods/d46cdd57-3bb7-4b44-863c-73c12433a40d/volumes" Oct 31 02:15:27.951206 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:27.951173 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:28.571383 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:28.571340 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:29.446249 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:29.441654 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:30.031880 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:30.031822 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:30.743670 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:30.743632 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:31.412503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:31.412469 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:32.108526 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:32.108480 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:32.488174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:32.488093 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:15:32.760660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:32.760624 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:33.283532 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:33.283480 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:33.422679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:33.422646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:33.569506 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:33.569422 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:33.768173 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:33.768078 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:34.319101 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:34.319060 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:34.790127 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:15:34.790091 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:15:35.053783 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:35.053621 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:35.795123 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:35.795079 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:36.432371 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:36.431584 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:37.072872 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:37.072818 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:37.782308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:37.782218 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:38.464773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:38.464626 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:39.345881 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:39.345844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:39.935116 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:39.935074 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:40.650596 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:40.650556 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:41.269322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:41.269287 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:41.906860 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:41.905212 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:42.571273 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:42.571238 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:43.251650 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:43.251615 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:43.837696 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:43.837667 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:44.471394 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:44.471365 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:44.606838 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:44.606798 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:45.063852 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:45.063763 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:45.846369 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:45.846316 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:46.525154 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:46.525122 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:47.208122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:47.208089 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:47.614874 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:47.614838 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9"] Oct 31 02:15:47.615339 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:47.615122 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" podUID="82aa30b9-8948-46ab-b29e-655807afa437" containerName="maas-api" containerID="cri-o://29d25db6f101a6e37c2e5d1de3066ace637d246f907d0d29cd120324aa69d068" gracePeriod=30 Oct 31 02:15:47.706320 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:47.706257 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:47.742351 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:47.742310 5090 generic.go:356] "Generic (PLEG): container finished" podID="82aa30b9-8948-46ab-b29e-655807afa437" containerID="29d25db6f101a6e37c2e5d1de3066ace637d246f907d0d29cd120324aa69d068" exitCode=0 Oct 31 02:15:47.749176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:47.749146 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" event={"ID":"82aa30b9-8948-46ab-b29e-655807afa437","Type":"ContainerDied","Data":"29d25db6f101a6e37c2e5d1de3066ace637d246f907d0d29cd120324aa69d068"} Oct 31 02:15:47.958751 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:47.957901 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:48.050484 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.050459 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 02:15:48.096414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.096379 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls\") pod \"82aa30b9-8948-46ab-b29e-655807afa437\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " Oct 31 02:15:48.096603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.096535 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82aa30b9-8948-46ab-b29e-655807afa437-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aa30b9-8948-46ab-b29e-655807afa437-openshift-service-ca\") pod \"82aa30b9-8948-46ab-b29e-655807afa437\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " Oct 31 02:15:48.096678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.096648 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82aa30b9-8948-46ab-b29e-655807afa437-kube-api-access-lkwfl\" (UniqueName: \"kubernetes.io/projected/82aa30b9-8948-46ab-b29e-655807afa437-kube-api-access-lkwfl\") pod \"82aa30b9-8948-46ab-b29e-655807afa437\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " Oct 31 02:15:48.096812 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.096791 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-metrics-certs\") pod \"82aa30b9-8948-46ab-b29e-655807afa437\" (UID: \"82aa30b9-8948-46ab-b29e-655807afa437\") " Oct 31 02:15:48.097041 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.097018 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82aa30b9-8948-46ab-b29e-655807afa437-openshift-service-ca" pod "82aa30b9-8948-46ab-b29e-655807afa437" (UID: "82aa30b9-8948-46ab-b29e-655807afa437"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:15:48.097332 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.097287 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aa30b9-8948-46ab-b29e-655807afa437-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:48.099796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.099770 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls" pod "82aa30b9-8948-46ab-b29e-655807afa437" (UID: "82aa30b9-8948-46ab-b29e-655807afa437"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:15:48.099890 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.099801 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82aa30b9-8948-46ab-b29e-655807afa437-kube-api-access-lkwfl" pod "82aa30b9-8948-46ab-b29e-655807afa437" (UID: "82aa30b9-8948-46ab-b29e-655807afa437"). InnerVolumeSpecName "kube-api-access-lkwfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:15:48.099950 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.099882 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-metrics-certs" pod "82aa30b9-8948-46ab-b29e-655807afa437" (UID: "82aa30b9-8948-46ab-b29e-655807afa437"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:15:48.146304 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.146227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:48.211962 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.204167 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkwfl\" (UniqueName: \"kubernetes.io/projected/82aa30b9-8948-46ab-b29e-655807afa437-kube-api-access-lkwfl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:48.211962 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.204218 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:48.211962 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.204234 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82aa30b9-8948-46ab-b29e-655807afa437-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:15:48.544210 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.544182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:48.751260 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.751233 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" Oct 31 02:15:48.751688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.751277 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9" event={"ID":"82aa30b9-8948-46ab-b29e-655807afa437","Type":"ContainerDied","Data":"72677fe51678e9d24227e52521b00644c88af88a897c2ee6d0fff77d4d8848c3"} Oct 31 02:15:48.751688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.751328 5090 scope.go:122] "RemoveContainer" containerID="29d25db6f101a6e37c2e5d1de3066ace637d246f907d0d29cd120324aa69d068" Oct 31 02:15:48.780420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.780395 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9"] Oct 31 02:15:48.784582 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.784554 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-e57add-6588c6f855-blmb9"] Oct 31 02:15:48.858091 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:48.856600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:49.653160 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:49.653116 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:49.757647 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:49.757365 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82aa30b9-8948-46ab-b29e-655807afa437" path="/var/lib/kubelet/pods/82aa30b9-8948-46ab-b29e-655807afa437/volumes" Oct 31 02:15:50.410224 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:50.409899 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:51.148011 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:51.147908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:51.815233 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:51.815195 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:52.311307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:52.311275 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:52.471951 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:52.471916 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:53.061591 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:53.061556 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:53.864256 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:53.864215 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:54.578847 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:54.578816 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:55.025185 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:55.025104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:55.161568 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:55.161530 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:55.304428 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:55.304351 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:55.452365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:55.452337 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:55.864920 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:55.864889 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:56.010629 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:56.010573 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:56.900232 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:56.900187 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:57.551846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:57.551806 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:58.185054 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:58.185016 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:58.931805 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:58.931766 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:15:59.555924 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:15:59.555889 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:00.273131 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:00.273101 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:00.932823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:00.932776 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:01.071493 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:01.071458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:01.751381 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:01.751320 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:02.510437 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:02.510401 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:03.246508 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:03.246450 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:04.460607 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:04.460571 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:04.596496 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:04.596460 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:04.790766 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:04.790709 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:16:05.146392 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:05.146301 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:05.945752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:05.945690 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:06.440785 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:06.440750 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:06.600166 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:06.600104 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:06.762663 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:06.762624 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:07.420406 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:07.420371 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:08.060677 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:08.060638 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:08.679037 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:08.679000 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:09.286299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:09.286253 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:09.935047 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:09.935015 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:10.642868 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:10.642835 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:11.353296 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:11.353246 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:12.061242 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:12.061195 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:12.767642 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:12.767606 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:13.358247 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:13.357075 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:14.003929 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:14.003899 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:14.352585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:14.352494 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:14.486670 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:14.486641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:14.635794 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:14.631971 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:15.373978 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:15.373849 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:16.214637 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:16.214602 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:17.050537 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:17.050487 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:17.454080 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:17.453984 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6698479b-klcb5"] Oct 31 02:16:17.513547 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:17.513517 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:17.665741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:17.665254 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:17.912890 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:17.912855 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:18.636921 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:18.636858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:18.988373 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:18.988289 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4"] Oct 31 02:16:19.058159 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:19.058124 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:19.216308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:19.216278 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:19.394086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:19.394053 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:19.588877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:19.588841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:19.749749 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:19.749701 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:20.038769 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:20.038648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:20.536524 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:20.536471 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:20.691639 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:20.691606 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:20.841847 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:20.840208 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:21.634964 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.634929 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:21.726193 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.726157 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v"] Oct 31 02:16:21.726817 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.726799 5090 cpu_manager.go:414] "containerMap: removing container" podUID="82aa30b9-8948-46ab-b29e-655807afa437" containerName="maas-api" Oct 31 02:16:21.726866 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.726820 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82aa30b9-8948-46ab-b29e-655807afa437" containerName="maas-api" Oct 31 02:16:21.726866 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.726845 5090 cpu_manager.go:414] "containerMap: removing container" podUID="d46cdd57-3bb7-4b44-863c-73c12433a40d" containerName="maas-api" Oct 31 02:16:21.726866 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.726850 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46cdd57-3bb7-4b44-863c-73c12433a40d" containerName="maas-api" Oct 31 02:16:21.726968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.726964 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="82aa30b9-8948-46ab-b29e-655807afa437" containerName="maas-api" Oct 31 02:16:21.727003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.726975 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46cdd57-3bb7-4b44-863c-73c12433a40d" containerName="maas-api" Oct 31 02:16:21.730528 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.730507 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.736566 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.736544 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"routing-overlay\"" Oct 31 02:16:21.739274 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.739252 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-20538a\"" Oct 31 02:16:21.739922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.739871 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"agc-cred-0a49e35b4edc\"" Oct 31 02:16:21.752141 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.752070 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v"] Oct 31 02:16:21.773083 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.773052 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:21.837615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.837584 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"openshift-service-ca.crt\"" Oct 31 02:16:21.837615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.837584 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-20538a\"/\"kube-root-ca.crt\"" Oct 31 02:16:21.849230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.849198 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vgrw\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-kube-api-access-6vgrw\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.849641 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.849254 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.849887 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.849827 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.850066 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.850043 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.951456 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.951361 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.951644 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.951472 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.951644 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.951603 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vgrw\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-kube-api-access-6vgrw\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.951856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.951696 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.952579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.952546 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.953227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.953197 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.955708 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.955681 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:21.970302 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:21.970278 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vgrw\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-kube-api-access-6vgrw\") pod \"payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:22.051105 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.051075 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:22.075697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.075666 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:22.160349 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.159341 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb"] Oct 31 02:16:22.168727 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.168671 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.172285 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.172215 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb"] Oct 31 02:16:22.173332 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.173305 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"agc-cred-7d8d23f33263\"" Oct 31 02:16:22.173856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.173796 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"kube-root-ca.crt\"" Oct 31 02:16:22.173856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.173810 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff\"" Oct 31 02:16:22.174058 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.173891 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"openshift-service-ca.crt\"" Oct 31 02:16:22.174163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.174147 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"payload-processing-external-model-e2e--4f663f8f-dockercfg-4bw6j\"" Oct 31 02:16:22.174374 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.174358 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d0c6ff\"/\"routing-overlay\"" Oct 31 02:16:22.279740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.273852 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.279740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.274005 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.279740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.274093 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zlm\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-kube-api-access-65zlm\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.279740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.274175 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.350083 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.349801 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v"] Oct 31 02:16:22.355678 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:16:22.355642 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495a37aa_6bcb_47cc_86f6_6d6ba48dbe05.slice/crio-0a6ad9e95611f75ff26f912c565c574cb9403b43c4ea14a1f9028aa2a88735cc WatchSource:0}: Error finding container 0a6ad9e95611f75ff26f912c565c574cb9403b43c4ea14a1f9028aa2a88735cc: Status 404 returned error can't find the container with id 0a6ad9e95611f75ff26f912c565c574cb9403b43c4ea14a1f9028aa2a88735cc Oct 31 02:16:22.375323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.375287 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.375455 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.375389 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.375535 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.375518 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.375613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.375596 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zlm\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-kube-api-access-65zlm\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.376383 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.376351 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.376488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.376380 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.378502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.378479 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.387903 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.387878 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zlm\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-kube-api-access-65zlm\") pod \"payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.455259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.455196 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" podUID="4aa20b91-0f02-481d-9e22-b5690eb78cf8" containerName="payload-processing" containerID="cri-o://a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c" gracePeriod=25 Oct 31 02:16:22.503033 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.503004 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:22.586838 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.574463 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:22.791820 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.791794 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:16:22.831449 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.831426 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb"] Oct 31 02:16:22.834407 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:16:22.834376 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4f1c2a_3b8d_49cc_91a0_202e959bc284.slice/crio-ccf23493b770e274b342703e7696d864aa225b3682597051c798f9b7b795b799 WatchSource:0}: Error finding container ccf23493b770e274b342703e7696d864aa225b3682597051c798f9b7b795b799: Status 404 returned error can't find the container with id ccf23493b770e274b342703e7696d864aa225b3682597051c798f9b7b795b799 Oct 31 02:16:22.882174 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.882150 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-provider-credentials\") pod \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " Oct 31 02:16:22.882486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.882259 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-routing-overlay-volume\") pod \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " Oct 31 02:16:22.882486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.882324 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-plugins-config-volume\") pod \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " Oct 31 02:16:22.882486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.882453 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-kube-api-access-mdxb7\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-kube-api-access-mdxb7\") pod \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\" (UID: \"4aa20b91-0f02-481d-9e22-b5690eb78cf8\") " Oct 31 02:16:22.882803 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.882771 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-routing-overlay-volume" pod "4aa20b91-0f02-481d-9e22-b5690eb78cf8" (UID: "4aa20b91-0f02-481d-9e22-b5690eb78cf8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:22.882901 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.882854 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-plugins-config-volume" pod "4aa20b91-0f02-481d-9e22-b5690eb78cf8" (UID: "4aa20b91-0f02-481d-9e22-b5690eb78cf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:22.883312 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.883293 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:22.883409 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.883320 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aa20b91-0f02-481d-9e22-b5690eb78cf8-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:22.884776 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.884753 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-provider-credentials" pod "4aa20b91-0f02-481d-9e22-b5690eb78cf8" (UID: "4aa20b91-0f02-481d-9e22-b5690eb78cf8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:22.885759 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.885698 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-kube-api-access-mdxb7" pod "4aa20b91-0f02-481d-9e22-b5690eb78cf8" (UID: "4aa20b91-0f02-481d-9e22-b5690eb78cf8"). InnerVolumeSpecName "kube-api-access-mdxb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:22.984594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.984568 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdxb7\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-kube-api-access-mdxb7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:22.984594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:22.984595 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4aa20b91-0f02-481d-9e22-b5690eb78cf8-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:23.018816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.018783 5090 generic.go:356] "Generic (PLEG): container finished" podID="4aa20b91-0f02-481d-9e22-b5690eb78cf8" containerID="a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c" exitCode=0 Oct 31 02:16:23.018936 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.018858 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" Oct 31 02:16:23.018936 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.018871 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" event={"ID":"4aa20b91-0f02-481d-9e22-b5690eb78cf8","Type":"ContainerDied","Data":"a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c"} Oct 31 02:16:23.018936 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.018927 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" event={"ID":"4aa20b91-0f02-481d-9e22-b5690eb78cf8","Type":"ContainerDied","Data":"86bd58b08844f3a01573b4568ebabaf6fc87de679ab37b3e8ee4ce0603387951"} Oct 31 02:16:23.019109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.018954 5090 scope.go:122] "RemoveContainer" containerID="a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c" Oct 31 02:16:23.020778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.020756 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" event={"ID":"6a4f1c2a-3b8d-49cc-91a0-202e959bc284","Type":"ContainerStarted","Data":"9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb"} Oct 31 02:16:23.020909 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.020815 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" event={"ID":"6a4f1c2a-3b8d-49cc-91a0-202e959bc284","Type":"ContainerStarted","Data":"ccf23493b770e274b342703e7696d864aa225b3682597051c798f9b7b795b799"} Oct 31 02:16:23.020909 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.020849 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:23.023180 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.023152 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" event={"ID":"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05","Type":"ContainerStarted","Data":"5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa"} Oct 31 02:16:23.023308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.023207 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" event={"ID":"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05","Type":"ContainerStarted","Data":"0a6ad9e95611f75ff26f912c565c574cb9403b43c4ea14a1f9028aa2a88735cc"} Oct 31 02:16:23.023308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.023237 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:23.029932 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.029763 5090 scope.go:122] "RemoveContainer" containerID="a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c" Oct 31 02:16:23.030395 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:23.030373 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c\": container with ID starting with a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c not found: ID does not exist" containerID="a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c" Oct 31 02:16:23.030481 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.030401 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c"} err="failed to get container status \"a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c\": rpc error: code = NotFound desc = could not find container \"a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c\": container with ID starting with a8491e15f9fe2f9682eedc0882a54ca73d185647c745e5077d20f6574665be8c not found: ID does not exist" Oct 31 02:16:23.042609 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.042561 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" podStartSLOduration=1.042544725 podStartE2EDuration="1.042544725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:23.039588494 +0000 UTC m=+3383.419833718" watchObservedRunningTime="2026-10-31 02:16:23.042544725 +0000 UTC m=+3383.422789944" Oct 31 02:16:23.058070 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.058027 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" podStartSLOduration=2.05800635 podStartE2EDuration="2.05800635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:16:23.056913447 +0000 UTC m=+3383.437158685" watchObservedRunningTime="2026-10-31 02:16:23.05800635 +0000 UTC m=+3383.438251572" Oct 31 02:16:23.070228 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.070204 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6698479b-klcb5"] Oct 31 02:16:23.080822 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.080799 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c6698479b-klcb5"] Oct 31 02:16:23.332259 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.332188 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:23.446904 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.446872 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v"] Oct 31 02:16:23.539636 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.539605 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:23.699807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.698569 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:23.730773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.726888 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6c6698479b-klcb5" podUID="4aa20b91-0f02-481d-9e22-b5690eb78cf8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.172:50052\" within 1s: context deadline exceeded" Oct 31 02:16:23.730773 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:16:23.727369 5090 logging.go:55] [core] [Channel #1424 SubChannel #1425]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.172:50052", ServerName: "10.132.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.172:50052: operation was canceled" Oct 31 02:16:23.752694 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.752659 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa20b91-0f02-481d-9e22-b5690eb78cf8" path="/var/lib/kubelet/pods/4aa20b91-0f02-481d-9e22-b5690eb78cf8/volumes" Oct 31 02:16:23.858054 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.858018 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb"] Oct 31 02:16:23.953140 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.953059 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:23.989314 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:23.989271 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" podUID="61dc5b10-f1f1-4eea-897a-b51407a7ef17" containerName="payload-processing" containerID="cri-o://a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a" gracePeriod=25 Oct 31 02:16:24.032003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.031978 5090 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" secret="" err="secret \"payload-processing-external-model-e2e--4f663f8f-dockercfg-4bw6j\" not found" Oct 31 02:16:24.157022 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.156986 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.226407 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-plugins-e2e-extmod-a-20538a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.226492 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:24.726469706 +0000 UTC m=+3385.106714911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227770 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d0c6ff/agc-cred-7d8d23f33263: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227789 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227835 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:24.727819177 +0000 UTC m=+3385.108064384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227874 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227908 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:24.727898415 +0000 UTC m=+3385.108143623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227959 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-20538a/agc-cred-0a49e35b4edc: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227968 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:24.230794 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.227995 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:24.727985752 +0000 UTC m=+3385.108230956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:24.320331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.320303 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:24.333311 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.333290 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:16:24.428154 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.428119 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-plugins-config-volume\") pod \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " Oct 31 02:16:24.428345 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.428319 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-routing-overlay-volume\") pod \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " Oct 31 02:16:24.428418 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.428354 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-provider-credentials\") pod \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " Oct 31 02:16:24.428481 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.428437 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-kube-api-access-8dtkb\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-kube-api-access-8dtkb\") pod \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\" (UID: \"61dc5b10-f1f1-4eea-897a-b51407a7ef17\") " Oct 31 02:16:24.428542 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.428492 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-plugins-config-volume" pod "61dc5b10-f1f1-4eea-897a-b51407a7ef17" (UID: "61dc5b10-f1f1-4eea-897a-b51407a7ef17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:24.428773 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.428747 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-routing-overlay-volume" pod "61dc5b10-f1f1-4eea-897a-b51407a7ef17" (UID: "61dc5b10-f1f1-4eea-897a-b51407a7ef17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:24.429119 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.429053 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:24.429119 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.429079 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61dc5b10-f1f1-4eea-897a-b51407a7ef17-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:24.430807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.430786 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-provider-credentials" pod "61dc5b10-f1f1-4eea-897a-b51407a7ef17" (UID: "61dc5b10-f1f1-4eea-897a-b51407a7ef17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:24.430930 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.430913 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-kube-api-access-8dtkb" pod "61dc5b10-f1f1-4eea-897a-b51407a7ef17" (UID: "61dc5b10-f1f1-4eea-897a-b51407a7ef17"). InnerVolumeSpecName "kube-api-access-8dtkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:24.530505 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.530472 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:24.530505 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.530505 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dtkb\" (UniqueName: \"kubernetes.io/projected/61dc5b10-f1f1-4eea-897a-b51407a7ef17-kube-api-access-8dtkb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:24.732616 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732579 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-plugins-e2e-extmod-a-20538a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732675 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732707 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d0c6ff/agc-cred-7d8d23f33263: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732705 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-20538a/agc-cred-0a49e35b4edc: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732740 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732749 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732739 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:25.732689716 +0000 UTC m=+3386.112934935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732799 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:25.732786413 +0000 UTC m=+3386.113031618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:24.732818 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732814 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:25.732805569 +0000 UTC m=+3386.113050771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:24.733099 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:24.732828 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:25.732820396 +0000 UTC m=+3386.113065622 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:24.867869 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:24.867767 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:25.039188 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.039154 5090 generic.go:356] "Generic (PLEG): container finished" podID="61dc5b10-f1f1-4eea-897a-b51407a7ef17" containerID="a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a" exitCode=0 Oct 31 02:16:25.039603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.039195 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" Oct 31 02:16:25.039603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.039232 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" event={"ID":"61dc5b10-f1f1-4eea-897a-b51407a7ef17","Type":"ContainerDied","Data":"a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a"} Oct 31 02:16:25.039603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.039289 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4" event={"ID":"61dc5b10-f1f1-4eea-897a-b51407a7ef17","Type":"ContainerDied","Data":"241a72d9e7f018264e3f9715b84f3b82692194868b65b701f9fe9d076737c2ba"} Oct 31 02:16:25.039603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.039315 5090 scope.go:122] "RemoveContainer" containerID="a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a" Oct 31 02:16:25.051221 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.051203 5090 scope.go:122] "RemoveContainer" containerID="a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a" Oct 31 02:16:25.051487 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.051468 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a\": container with ID starting with a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a not found: ID does not exist" containerID="a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a" Oct 31 02:16:25.051538 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.051497 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a"} err="failed to get container status \"a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a\": rpc error: code = NotFound desc = could not find container \"a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a\": container with ID starting with a3848b0b080036779883d964b858caffdfa1ab5bc041b9a4973ef9dfe9e9249a not found: ID does not exist" Oct 31 02:16:25.092299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.092275 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4"] Oct 31 02:16:25.096522 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.096500 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-91b6b1/payload-processing-external-model-e2e-extmod-b-91b6b1-57b7v99v4"] Oct 31 02:16:25.547024 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.546991 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:25.747587 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:25.747556 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61dc5b10-f1f1-4eea-897a-b51407a7ef17" path="/var/lib/kubelet/pods/61dc5b10-f1f1-4eea-897a-b51407a7ef17/volumes" Oct 31 02:16:25.763503 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763474 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d0c6ff/agc-cred-7d8d23f33263: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:25.763503 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763497 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:25.763658 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763508 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:25.763658 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763543 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:27.763529232 +0000 UTC m=+3388.143774436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:25.763658 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763558 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:27.763551416 +0000 UTC m=+3388.143796617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:25.763658 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763603 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-20538a/agc-cred-0a49e35b4edc: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:25.763658 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763615 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:25.763658 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763644 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:27.763632087 +0000 UTC m=+3388.143877288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:25.763899 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763772 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-plugins-e2e-extmod-a-20538a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:25.763899 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:25.763818 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:27.763807751 +0000 UTC m=+3388.144052953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:26.232086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:26.231734 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:26.956697 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:26.956665 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:27.616276 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:27.612938 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:27.803466 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.803435 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-plugins-e2e-extmod-a-20538a: configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:27.803666 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.803524 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:31.8035086 +0000 UTC m=+3392.183753818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-20538a" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.803913 5090 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff: configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.803966 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d0c6ff/agc-cred-7d8d23f33263: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.803988 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb: secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.804027 5090 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-20538a/agc-cred-0a49e35b4edc: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.804037 5090 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v: secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.804039 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:31.804020093 +0000 UTC m=+3392.184265314 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-d0c6ff" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.804090 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials podName:6a4f1c2a-3b8d-49cc-91a0-202e959bc284 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:31.804075247 +0000 UTC m=+3392.184320465 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284") : secret "agc-cred-7d8d23f33263" not found Oct 31 02:16:27.804240 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:27.804108 5090 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials podName:495a37aa-6bcb-47cc-86f6-6d6ba48dbe05 nodeName:}" failed. No retries permitted until 2026-10-31 02:16:31.804099319 +0000 UTC m=+3392.184344522 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05") : secret "agc-cred-0a49e35b4edc" not found Oct 31 02:16:28.343555 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:28.343523 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:29.060764 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:29.060725 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:29.872572 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:29.872537 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:30.040888 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:30.040816 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" podUID="495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" containerName="payload-processing" containerID="cri-o://5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa" gracePeriod=25 Oct 31 02:16:30.041197 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:30.041163 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" podUID="6a4f1c2a-3b8d-49cc-91a0-202e959bc284" containerName="payload-processing" containerID="cri-o://9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb" gracePeriod=25 Oct 31 02:16:30.869464 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:30.869432 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:30.886736 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:30.876871 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:31.008295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:30.999969 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-kube-api-access-6vgrw\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-kube-api-access-6vgrw\") pod \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " Oct 31 02:16:31.008295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.000039 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-kube-api-access-65zlm\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-kube-api-access-65zlm\") pod \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " Oct 31 02:16:31.008295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.000094 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-routing-overlay-volume\") pod \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " Oct 31 02:16:31.008295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.001621 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-routing-overlay-volume" pod "6a4f1c2a-3b8d-49cc-91a0-202e959bc284" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:31.093122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.089280 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.109267 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials\") pod \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.109386 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials\") pod \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.109434 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume\") pod \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\" (UID: \"6a4f1c2a-3b8d-49cc-91a0-202e959bc284\") " Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.109487 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume\") pod \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.109520 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-routing-overlay-volume\") pod \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\" (UID: \"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05\") " Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.110164 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.110549 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-routing-overlay-volume" pod "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:31.131742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.122198 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume" pod "6a4f1c2a-3b8d-49cc-91a0-202e959bc284" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:31.146075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.146032 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume" pod "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:31.200209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.200176 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials" pod "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:31.200940 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.200900 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials" pod "6a4f1c2a-3b8d-49cc-91a0-202e959bc284" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:31.231156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.216800 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.231156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.216831 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.231156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.216848 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.231156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.216863 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.231156 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.216878 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.247355 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.247050 5090 generic.go:356] "Generic (PLEG): container finished" podID="495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" containerID="5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa" exitCode=0 Oct 31 02:16:31.247355 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.247312 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" event={"ID":"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05","Type":"ContainerDied","Data":"5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa"} Oct 31 02:16:31.247615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.247403 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" event={"ID":"495a37aa-6bcb-47cc-86f6-6d6ba48dbe05","Type":"ContainerDied","Data":"0a6ad9e95611f75ff26f912c565c574cb9403b43c4ea14a1f9028aa2a88735cc"} Oct 31 02:16:31.247615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.247454 5090 scope.go:122] "RemoveContainer" containerID="5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa" Oct 31 02:16:31.247783 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.247617 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v" Oct 31 02:16:31.282740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.282644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:31.309401 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.298304 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-kube-api-access-65zlm" pod "6a4f1c2a-3b8d-49cc-91a0-202e959bc284" (UID: "6a4f1c2a-3b8d-49cc-91a0-202e959bc284"). InnerVolumeSpecName "kube-api-access-65zlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:31.334739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.311454 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-kube-api-access-6vgrw" pod "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" (UID: "495a37aa-6bcb-47cc-86f6-6d6ba48dbe05"). InnerVolumeSpecName "kube-api-access-6vgrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:31.334739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.332258 5090 generic.go:356] "Generic (PLEG): container finished" podID="6a4f1c2a-3b8d-49cc-91a0-202e959bc284" containerID="9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb" exitCode=0 Oct 31 02:16:31.334739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.332326 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" Oct 31 02:16:31.334739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.332371 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" event={"ID":"6a4f1c2a-3b8d-49cc-91a0-202e959bc284","Type":"ContainerDied","Data":"9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb"} Oct 31 02:16:31.334739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.332420 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb" event={"ID":"6a4f1c2a-3b8d-49cc-91a0-202e959bc284","Type":"ContainerDied","Data":"ccf23493b770e274b342703e7696d864aa225b3682597051c798f9b7b795b799"} Oct 31 02:16:31.334739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.334381 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vgrw\" (UniqueName: \"kubernetes.io/projected/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05-kube-api-access-6vgrw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.334739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.334411 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zlm\" (UniqueName: \"kubernetes.io/projected/6a4f1c2a-3b8d-49cc-91a0-202e959bc284-kube-api-access-65zlm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:31.340617 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.340490 5090 scope.go:122] "RemoveContainer" containerID="5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa" Oct 31 02:16:31.341413 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:31.341260 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa\": container with ID starting with 5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa not found: ID does not exist" containerID="5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa" Oct 31 02:16:31.341413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.341300 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa"} err="failed to get container status \"5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa\": rpc error: code = NotFound desc = could not find container \"5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa\": container with ID starting with 5ac872e8c03846bc86730448e3f5aa272eb5bd7e71ba240c5f14f1335111fffa not found: ID does not exist" Oct 31 02:16:31.341413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.341327 5090 scope.go:122] "RemoveContainer" containerID="9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb" Oct 31 02:16:31.374520 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.371793 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb"] Oct 31 02:16:31.380311 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.380281 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d0c6ff/payload-processing-external-model-e2e-extmod-b-d0c6ff-778cp2rcb"] Oct 31 02:16:31.395497 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.395149 5090 scope.go:122] "RemoveContainer" containerID="9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb" Oct 31 02:16:31.395829 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:31.395729 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb\": container with ID starting with 9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb not found: ID does not exist" containerID="9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb" Oct 31 02:16:31.395829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.395767 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb"} err="failed to get container status \"9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb\": rpc error: code = NotFound desc = could not find container \"9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb\": container with ID starting with 9bebab1b5f71f71e13bca206cf05a0a598eed1c694214e2bc7e4ff3e466917cb not found: ID does not exist" Oct 31 02:16:31.444039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.443998 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5"] Oct 31 02:16:31.554692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.554403 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:31.623844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.623806 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v"] Oct 31 02:16:31.626183 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.626135 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-20538a/payload-processing-external-model-e2e-extmod-a-20538a-6bd6mx28v"] Oct 31 02:16:31.790919 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.790809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:31.797183 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.797153 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" path="/var/lib/kubelet/pods/495a37aa-6bcb-47cc-86f6-6d6ba48dbe05/volumes" Oct 31 02:16:31.797994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:31.797967 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4f1c2a-3b8d-49cc-91a0-202e959bc284" path="/var/lib/kubelet/pods/6a4f1c2a-3b8d-49cc-91a0-202e959bc284/volumes" Oct 31 02:16:32.475611 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:32.475578 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:33.170879 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:33.170841 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:33.338486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:33.338456 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:34.087253 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:34.087204 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:34.634693 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:34.634660 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx"] Oct 31 02:16:34.634986 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:34.634908 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" podUID="56fa5427-486c-496c-b778-518435b1e0d3" containerName="maas-api" containerID="cri-o://7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854" gracePeriod=30 Oct 31 02:16:34.721750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:34.721681 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:34.802769 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:34.800508 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:16:34.922793 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:34.913534 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:35.057678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.057653 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 02:16:35.138884 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.138853 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:35.139974 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.139942 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56fa5427-486c-496c-b778-518435b1e0d3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5427-486c-496c-b778-518435b1e0d3-openshift-service-ca\") pod \"56fa5427-486c-496c-b778-518435b1e0d3\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " Oct 31 02:16:35.140128 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.140003 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls\") pod \"56fa5427-486c-496c-b778-518435b1e0d3\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " Oct 31 02:16:35.140128 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.140072 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-metrics-certs\") pod \"56fa5427-486c-496c-b778-518435b1e0d3\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " Oct 31 02:16:35.140660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.140333 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56fa5427-486c-496c-b778-518435b1e0d3-kube-api-access-6l26f\" (UniqueName: \"kubernetes.io/projected/56fa5427-486c-496c-b778-518435b1e0d3-kube-api-access-6l26f\") pod \"56fa5427-486c-496c-b778-518435b1e0d3\" (UID: \"56fa5427-486c-496c-b778-518435b1e0d3\") " Oct 31 02:16:35.140848 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.140683 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56fa5427-486c-496c-b778-518435b1e0d3-openshift-service-ca" pod "56fa5427-486c-496c-b778-518435b1e0d3" (UID: "56fa5427-486c-496c-b778-518435b1e0d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:35.145945 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.145917 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56fa5427-486c-496c-b778-518435b1e0d3-kube-api-access-6l26f" pod "56fa5427-486c-496c-b778-518435b1e0d3" (UID: "56fa5427-486c-496c-b778-518435b1e0d3"). InnerVolumeSpecName "kube-api-access-6l26f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:35.147444 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.147422 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-metrics-certs" pod "56fa5427-486c-496c-b778-518435b1e0d3" (UID: "56fa5427-486c-496c-b778-518435b1e0d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:16:35.147564 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.147501 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls" pod "56fa5427-486c-496c-b778-518435b1e0d3" (UID: "56fa5427-486c-496c-b778-518435b1e0d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:16:35.241679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.241596 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l26f\" (UniqueName: \"kubernetes.io/projected/56fa5427-486c-496c-b778-518435b1e0d3-kube-api-access-6l26f\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:35.241679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.241636 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56fa5427-486c-496c-b778-518435b1e0d3-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:35.241679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.241652 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:35.241679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.241665 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56fa5427-486c-496c-b778-518435b1e0d3-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:35.331549 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.331505 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:35.376913 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.376877 5090 generic.go:356] "Generic (PLEG): container finished" podID="56fa5427-486c-496c-b778-518435b1e0d3" containerID="7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854" exitCode=0 Oct 31 02:16:35.377114 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.376949 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" Oct 31 02:16:35.377114 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.376969 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" event={"ID":"56fa5427-486c-496c-b778-518435b1e0d3","Type":"ContainerDied","Data":"7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854"} Oct 31 02:16:35.377114 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.377043 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx" event={"ID":"56fa5427-486c-496c-b778-518435b1e0d3","Type":"ContainerDied","Data":"b3ecc7fd6a366a4f71f0cf736974a8a2e7af8dd4a3f1da290f250f2c91129e15"} Oct 31 02:16:35.377114 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.377074 5090 scope.go:122] "RemoveContainer" containerID="7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854" Oct 31 02:16:35.389391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.389373 5090 scope.go:122] "RemoveContainer" containerID="7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854" Oct 31 02:16:35.389644 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:35.389624 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854\": container with ID starting with 7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854 not found: ID does not exist" containerID="7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854" Oct 31 02:16:35.389752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.389651 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854"} err="failed to get container status \"7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854\": rpc error: code = NotFound desc = could not find container \"7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854\": container with ID starting with 7c785c128737fa1dc5b61f94a606b968b57ee813ba8bee12b77b3764a57a7854 not found: ID does not exist" Oct 31 02:16:35.406966 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.406939 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx"] Oct 31 02:16:35.413337 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.413317 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-3cf6f4-b8f5587b5-qqgqx"] Oct 31 02:16:35.668004 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.667974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:35.747462 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.747434 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56fa5427-486c-496c-b778-518435b1e0d3" path="/var/lib/kubelet/pods/56fa5427-486c-496c-b778-518435b1e0d3/volumes" Oct 31 02:16:35.888998 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:35.888969 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:36.445973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.445927 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" podUID="3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" containerName="payload-processing" containerID="cri-o://1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d" gracePeriod=25 Oct 31 02:16:36.662532 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.662345 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:36.750114 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.750088 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:16:36.764399 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.764363 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-routing-overlay-volume\") pod \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " Oct 31 02:16:36.764519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.764495 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-plugins-config-volume\") pod \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " Oct 31 02:16:36.764576 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.764545 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-kube-api-access-c9xb6\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-kube-api-access-c9xb6\") pod \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " Oct 31 02:16:36.764688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.764669 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-provider-credentials\") pod \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\" (UID: \"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b\") " Oct 31 02:16:36.764866 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.764756 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-routing-overlay-volume" pod "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" (UID: "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:36.764984 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.764964 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-plugins-config-volume" pod "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" (UID: "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:36.765184 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.765167 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:36.765254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.765192 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:36.767459 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.767437 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-provider-credentials" pod "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" (UID: "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:36.767525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.767469 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-kube-api-access-c9xb6" pod "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" (UID: "3968f3a1-4bc1-4e90-a4b2-6d1542d6353b"). InnerVolumeSpecName "kube-api-access-c9xb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:36.866032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.865995 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xb6\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-kube-api-access-c9xb6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:36.866032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:36.866027 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:37.391912 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.391875 5090 generic.go:356] "Generic (PLEG): container finished" podID="3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" containerID="1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d" exitCode=0 Oct 31 02:16:37.392117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.391922 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" Oct 31 02:16:37.392117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.391940 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" event={"ID":"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b","Type":"ContainerDied","Data":"1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d"} Oct 31 02:16:37.392117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.391994 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5" event={"ID":"3968f3a1-4bc1-4e90-a4b2-6d1542d6353b","Type":"ContainerDied","Data":"46a98f5cde2963d8fc6b6b62280178f7d756e6d0a54efb2263c5e38dce6de812"} Oct 31 02:16:37.392117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.392015 5090 scope.go:122] "RemoveContainer" containerID="1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d" Oct 31 02:16:37.403007 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.402985 5090 scope.go:122] "RemoveContainer" containerID="1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d" Oct 31 02:16:37.403270 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:37.403249 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d\": container with ID starting with 1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d not found: ID does not exist" containerID="1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d" Oct 31 02:16:37.403317 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.403280 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d"} err="failed to get container status \"1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d\": rpc error: code = NotFound desc = could not find container \"1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d\": container with ID starting with 1f71e9967b04805886e450e29e60a66a5bd1a08fe729df22809a172649520c2d not found: ID does not exist" Oct 31 02:16:37.417267 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.417244 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5"] Oct 31 02:16:37.428366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.428305 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bb5c51/payload-processing-external-model-e2e-extmod-a-bb5c51-5bf4rjpg5"] Oct 31 02:16:37.438204 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.438125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:37.746540 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:37.746510 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" path="/var/lib/kubelet/pods/3968f3a1-4bc1-4e90-a4b2-6d1542d6353b/volumes" Oct 31 02:16:38.077031 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:38.076921 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:38.902852 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:38.902815 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:39.565595 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:39.565559 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:39.710967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:39.710938 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:39.859852 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:39.859762 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:40.473603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:40.473569 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:40.884121 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:40.884084 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:41.637555 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:41.637520 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:42.356826 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:42.356794 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:42.519008 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:42.518958 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:42.673555 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:42.673463 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc"] Oct 31 02:16:42.674357 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:42.674322 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" containerID="cri-o://2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d" gracePeriod=30 Oct 31 02:16:42.738973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:42.738937 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:43.260212 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:43.260171 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:43.772732 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:43.772684 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:43.911853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:43.911787 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:44.274700 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:44.274664 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:44.978146 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:44.978092 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:45.546136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:45.546088 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:45.685045 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:45.684991 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:45.826125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:45.826026 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:45.965615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:45.965561 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:46.337231 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:46.337163 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:46.755689 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:46.755543 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:47.027479 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:47.027403 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:47.510808 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:47.510703 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:48.252050 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.252017 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:48.691151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.691121 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 02:16:48.881046 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.880961 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-credential-socket\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881046 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881041 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-socket\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881257 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881157 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-data\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881257 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881221 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-podinfo\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881257 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881238 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-credential-socket" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:16:48.881450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881280 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-token\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-token\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881300 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-socket" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:16:48.881450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881358 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-certs\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881427 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istiod-ca-cert\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881450 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881432 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-data" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:16:48.881758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881467 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-envoy\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881506 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-kube-api-access-xt468\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-kube-api-access-xt468\") pod \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\" (UID: \"fcb06cda-3663-461c-93ca-7f63f7c12aa1\") " Oct 31 02:16:48.881758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881638 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-certs" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:16:48.881947 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.881791 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istiod-ca-cert" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:48.882104 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.882080 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.882216 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.882108 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.882216 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.882124 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.882216 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.882139 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.882216 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.882191 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.883904 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.883851 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-podinfo" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:16:48.884047 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.884027 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-kube-api-access-xt468" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "kube-api-access-xt468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:48.884261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.884242 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-token" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:48.884323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.884262 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-envoy" pod "fcb06cda-3663-461c-93ca-7f63f7c12aa1" (UID: "fcb06cda-3663-461c-93ca-7f63f7c12aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:16:48.914758 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.914458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:48.982960 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.982919 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.982960 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.982952 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.982960 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.982965 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcb06cda-3663-461c-93ca-7f63f7c12aa1-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:48.983200 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:48.982980 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt468\" (UniqueName: \"kubernetes.io/projected/fcb06cda-3663-461c-93ca-7f63f7c12aa1-kube-api-access-xt468\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:49.475706 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.475666 5090 generic.go:356] "Generic (PLEG): container finished" podID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerID="2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d" exitCode=0 Oct 31 02:16:49.475706 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.475736 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" Oct 31 02:16:49.476003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.475752 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" event={"ID":"fcb06cda-3663-461c-93ca-7f63f7c12aa1","Type":"ContainerDied","Data":"2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d"} Oct 31 02:16:49.476003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.475813 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc" event={"ID":"fcb06cda-3663-461c-93ca-7f63f7c12aa1","Type":"ContainerDied","Data":"68b490c144ec11331268f53e6690e58d6bd77e9a01e6510d8cd3957c0d74d50d"} Oct 31 02:16:49.476003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.475841 5090 scope.go:122] "RemoveContainer" containerID="2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d" Oct 31 02:16:49.490811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.490791 5090 scope.go:122] "RemoveContainer" containerID="2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d" Oct 31 02:16:49.491049 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:49.491030 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d\": container with ID starting with 2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d not found: ID does not exist" containerID="2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d" Oct 31 02:16:49.491089 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.491059 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d"} err="failed to get container status \"2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d\": rpc error: code = NotFound desc = could not find container \"2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d\": container with ID starting with 2c2f7320d99dbdb9b3863ae99e536f411d1cc3ab4dd7afb4bb073e1e384f420d not found: ID does not exist" Oct 31 02:16:49.505201 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.505177 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc"] Oct 31 02:16:49.510022 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.509991 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3cf6f4-openshift-default-64cc779d9c-t5fjc"] Oct 31 02:16:49.560958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.560597 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:49.747120 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:49.747092 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" path="/var/lib/kubelet/pods/fcb06cda-3663-461c-93ca-7f63f7c12aa1/volumes" Oct 31 02:16:50.136549 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:50.136471 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:50.340117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:50.340080 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:50.837132 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:50.834600 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:51.475488 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:51.475434 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:52.087014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:52.086975 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:52.360463 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:52.360391 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:52.765761 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:52.760470 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:53.216534 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:53.216445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:53.361550 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:53.361498 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:53.513269 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:53.508407 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:53.650743 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:53.649733 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:54.305199 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:54.304568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:55.131545 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:55.131513 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:55.980317 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:55.980273 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:56.663119 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:56.663089 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:57.552684 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:57.552646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:58.211973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.211939 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl"] Oct 31 02:16:58.319825 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.319735 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:58.476795 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.468176 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:58.648752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.648700 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj"] Oct 31 02:16:58.743509 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.743485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:58.856374 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.856338 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66"] Oct 31 02:16:58.856619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.856575 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" podUID="56101a31-a9b1-4b1d-9214-20d1ac327da9" containerName="maas-api" containerID="cri-o://f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8" gracePeriod=30 Oct 31 02:16:58.938194 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:58.938163 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:59.174907 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.174829 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:59.253554 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.253491 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:16:59.263395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.263349 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56101a31-a9b1-4b1d-9214-20d1ac327da9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56101a31-a9b1-4b1d-9214-20d1ac327da9-openshift-service-ca\") pod \"56101a31-a9b1-4b1d-9214-20d1ac327da9\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " Oct 31 02:16:59.263565 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.263403 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-metrics-certs\") pod \"56101a31-a9b1-4b1d-9214-20d1ac327da9\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " Oct 31 02:16:59.263671 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.263653 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56101a31-a9b1-4b1d-9214-20d1ac327da9-kube-api-access-2hw5m\" (UniqueName: \"kubernetes.io/projected/56101a31-a9b1-4b1d-9214-20d1ac327da9-kube-api-access-2hw5m\") pod \"56101a31-a9b1-4b1d-9214-20d1ac327da9\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " Oct 31 02:16:59.263886 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.263854 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56101a31-a9b1-4b1d-9214-20d1ac327da9-openshift-service-ca" pod "56101a31-a9b1-4b1d-9214-20d1ac327da9" (UID: "56101a31-a9b1-4b1d-9214-20d1ac327da9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:16:59.264473 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.264451 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls\") pod \"56101a31-a9b1-4b1d-9214-20d1ac327da9\" (UID: \"56101a31-a9b1-4b1d-9214-20d1ac327da9\") " Oct 31 02:16:59.265222 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.265192 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56101a31-a9b1-4b1d-9214-20d1ac327da9-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:59.266476 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.266454 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-metrics-certs" pod "56101a31-a9b1-4b1d-9214-20d1ac327da9" (UID: "56101a31-a9b1-4b1d-9214-20d1ac327da9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:16:59.266608 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.266577 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56101a31-a9b1-4b1d-9214-20d1ac327da9-kube-api-access-2hw5m" pod "56101a31-a9b1-4b1d-9214-20d1ac327da9" (UID: "56101a31-a9b1-4b1d-9214-20d1ac327da9"). InnerVolumeSpecName "kube-api-access-2hw5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:16:59.267118 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.267094 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls" pod "56101a31-a9b1-4b1d-9214-20d1ac327da9" (UID: "56101a31-a9b1-4b1d-9214-20d1ac327da9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:16:59.322736 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.322691 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:59.366050 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.366022 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hw5m\" (UniqueName: \"kubernetes.io/projected/56101a31-a9b1-4b1d-9214-20d1ac327da9-kube-api-access-2hw5m\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:59.366050 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.366048 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:59.366242 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.366058 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56101a31-a9b1-4b1d-9214-20d1ac327da9-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:16:59.466929 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.466895 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:16:59.556014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.555934 5090 generic.go:356] "Generic (PLEG): container finished" podID="56101a31-a9b1-4b1d-9214-20d1ac327da9" containerID="f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8" exitCode=0 Oct 31 02:16:59.556014 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.555973 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" Oct 31 02:16:59.556214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.555975 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" event={"ID":"56101a31-a9b1-4b1d-9214-20d1ac327da9","Type":"ContainerDied","Data":"f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8"} Oct 31 02:16:59.556214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.556108 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66" event={"ID":"56101a31-a9b1-4b1d-9214-20d1ac327da9","Type":"ContainerDied","Data":"16a32f4141e9259e49afb3b57dcd72ab88c3f25c0def9391b17e7a4a831ba9ad"} Oct 31 02:16:59.556214 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.556136 5090 scope.go:122] "RemoveContainer" containerID="f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8" Oct 31 02:16:59.566967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.566943 5090 scope.go:122] "RemoveContainer" containerID="f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8" Oct 31 02:16:59.567256 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:16:59.567234 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8\": container with ID starting with f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8 not found: ID does not exist" containerID="f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8" Oct 31 02:16:59.567339 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.567269 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8"} err="failed to get container status \"f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8\": rpc error: code = NotFound desc = could not find container \"f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8\": container with ID starting with f33b4903bc9a75f53600660f7d9e95dec581a7112a8b8bc0abab04852c1e42d8 not found: ID does not exist" Oct 31 02:16:59.586303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.586278 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66"] Oct 31 02:16:59.588992 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.588972 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bb5c51-547f7fb558-mgv66"] Oct 31 02:16:59.746807 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.746772 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56101a31-a9b1-4b1d-9214-20d1ac327da9" path="/var/lib/kubelet/pods/56101a31-a9b1-4b1d-9214-20d1ac327da9/volumes" Oct 31 02:16:59.820358 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:16:59.820287 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:00.177354 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:00.177258 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:00.842424 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:00.842388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:01.465967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:01.465931 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:02.159603 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:02.159563 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:02.836589 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:02.836554 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:03.212854 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.212751 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" podUID="8a83099b-12c5-403d-8695-923fdce5db09" containerName="payload-processing" containerID="cri-o://6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0" gracePeriod=25 Oct 31 02:17:03.479595 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.479538 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:17:03.511685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.509191 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-provider-credentials\") pod \"8a83099b-12c5-403d-8695-923fdce5db09\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " Oct 31 02:17:03.511685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.509353 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-routing-overlay-volume\") pod \"8a83099b-12c5-403d-8695-923fdce5db09\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " Oct 31 02:17:03.511685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.510095 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-plugins-config-volume\") pod \"8a83099b-12c5-403d-8695-923fdce5db09\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " Oct 31 02:17:03.511685 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.510336 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-kube-api-access-4tg29\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-kube-api-access-4tg29\") pod \"8a83099b-12c5-403d-8695-923fdce5db09\" (UID: \"8a83099b-12c5-403d-8695-923fdce5db09\") " Oct 31 02:17:03.516395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.515272 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-routing-overlay-volume" pod "8a83099b-12c5-403d-8695-923fdce5db09" (UID: "8a83099b-12c5-403d-8695-923fdce5db09"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:03.516395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.516046 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-plugins-config-volume" pod "8a83099b-12c5-403d-8695-923fdce5db09" (UID: "8a83099b-12c5-403d-8695-923fdce5db09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:03.517029 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.516947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:03.522176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.520219 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-provider-credentials" pod "8a83099b-12c5-403d-8695-923fdce5db09" (UID: "8a83099b-12c5-403d-8695-923fdce5db09"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:03.525127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.525086 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-kube-api-access-4tg29" pod "8a83099b-12c5-403d-8695-923fdce5db09" (UID: "8a83099b-12c5-403d-8695-923fdce5db09"). InnerVolumeSpecName "kube-api-access-4tg29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:03.581856 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.581820 5090 generic.go:356] "Generic (PLEG): container finished" podID="8a83099b-12c5-403d-8695-923fdce5db09" containerID="6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0" exitCode=0 Oct 31 02:17:03.582039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.581871 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" Oct 31 02:17:03.582039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.581901 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" event={"ID":"8a83099b-12c5-403d-8695-923fdce5db09","Type":"ContainerDied","Data":"6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0"} Oct 31 02:17:03.582039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.581973 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl" event={"ID":"8a83099b-12c5-403d-8695-923fdce5db09","Type":"ContainerDied","Data":"29bff9a09701a2999a90f31db784952209c22c5337215e38fa967ed0054e5cfe"} Oct 31 02:17:03.582039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.582003 5090 scope.go:122] "RemoveContainer" containerID="6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0" Oct 31 02:17:03.592844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.592823 5090 scope.go:122] "RemoveContainer" containerID="6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0" Oct 31 02:17:03.593075 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:03.593058 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0\": container with ID starting with 6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0 not found: ID does not exist" containerID="6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0" Oct 31 02:17:03.593133 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.593084 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0"} err="failed to get container status \"6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0\": rpc error: code = NotFound desc = could not find container \"6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0\": container with ID starting with 6cb48dadf25a6a81c5d9ff5916b2110041f488aecc36ad9787036a588cec41f0 not found: ID does not exist" Oct 31 02:17:03.607149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.607124 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl"] Oct 31 02:17:03.611781 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.611756 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:03.611888 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.611783 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tg29\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-kube-api-access-4tg29\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:03.611888 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.611798 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8a83099b-12c5-403d-8695-923fdce5db09-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:03.611888 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.611811 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8a83099b-12c5-403d-8695-923fdce5db09-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:03.613804 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.613784 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-28e4f3/payload-processing-external-model-e2e-extmod-a-28e4f3-597dd94wl"] Oct 31 02:17:03.649695 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.649667 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" podUID="3ca05e63-f923-4eec-8c2e-f52416ba67f4" containerName="payload-processing" containerID="cri-o://acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09" gracePeriod=25 Oct 31 02:17:03.746125 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.746100 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a83099b-12c5-403d-8695-923fdce5db09" path="/var/lib/kubelet/pods/8a83099b-12c5-403d-8695-923fdce5db09/volumes" Oct 31 02:17:03.896619 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.896593 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:17:03.914646 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.914615 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-kube-api-access-mdth5\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-kube-api-access-mdth5\") pod \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " Oct 31 02:17:03.914823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.914754 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-provider-credentials\") pod \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " Oct 31 02:17:03.914823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.914797 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-plugins-config-volume\") pod \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " Oct 31 02:17:03.915021 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.914831 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-routing-overlay-volume\") pod \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\" (UID: \"3ca05e63-f923-4eec-8c2e-f52416ba67f4\") " Oct 31 02:17:03.915308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.915277 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-routing-overlay-volume" pod "3ca05e63-f923-4eec-8c2e-f52416ba67f4" (UID: "3ca05e63-f923-4eec-8c2e-f52416ba67f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:03.915308 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.915293 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-plugins-config-volume" pod "3ca05e63-f923-4eec-8c2e-f52416ba67f4" (UID: "3ca05e63-f923-4eec-8c2e-f52416ba67f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:03.915693 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.915667 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:03.915693 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.915694 5090 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3ca05e63-f923-4eec-8c2e-f52416ba67f4-routing-overlay-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:03.918374 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.918347 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-provider-credentials" pod "3ca05e63-f923-4eec-8c2e-f52416ba67f4" (UID: "3ca05e63-f923-4eec-8c2e-f52416ba67f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:03.919561 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:03.919536 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-kube-api-access-mdth5" pod "3ca05e63-f923-4eec-8c2e-f52416ba67f4" (UID: "3ca05e63-f923-4eec-8c2e-f52416ba67f4"). InnerVolumeSpecName "kube-api-access-mdth5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:04.016628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.016555 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdth5\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-kube-api-access-mdth5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:04.016628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.016583 5090 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3ca05e63-f923-4eec-8c2e-f52416ba67f4-provider-credentials\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:04.242155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.242085 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:04.397660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.397584 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:04.545436 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.545341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:04.616253 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.616214 5090 generic.go:356] "Generic (PLEG): container finished" podID="3ca05e63-f923-4eec-8c2e-f52416ba67f4" containerID="acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09" exitCode=0 Oct 31 02:17:04.616478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.616360 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" Oct 31 02:17:04.616966 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.616929 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" event={"ID":"3ca05e63-f923-4eec-8c2e-f52416ba67f4","Type":"ContainerDied","Data":"acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09"} Oct 31 02:17:04.617126 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.616985 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj" event={"ID":"3ca05e63-f923-4eec-8c2e-f52416ba67f4","Type":"ContainerDied","Data":"11397ea8d3606b1e8dc0fb261896dec03dbcd00901916414adcd139ca16c062b"} Oct 31 02:17:04.617126 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.617017 5090 scope.go:122] "RemoveContainer" containerID="acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09" Oct 31 02:17:04.637009 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.636982 5090 scope.go:122] "RemoveContainer" containerID="acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09" Oct 31 02:17:04.637487 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:04.637459 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09\": container with ID starting with acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09 not found: ID does not exist" containerID="acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09" Oct 31 02:17:04.637591 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.637499 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09"} err="failed to get container status \"acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09\": rpc error: code = NotFound desc = could not find container \"acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09\": container with ID starting with acf879df1265080be0dd990d1e9dc3ee28d5a6593c062c2cc3f3b7fa11441f09 not found: ID does not exist" Oct 31 02:17:04.649971 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.649920 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj"] Oct 31 02:17:04.655502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.655479 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c088fd/payload-processing-external-model-e2e-extmod-b-c088fd-6bdcmkgpj"] Oct 31 02:17:04.789311 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:04.789276 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:17:04.962973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:04.962883 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:05.598039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:05.597994 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:05.746164 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:05.746134 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca05e63-f923-4eec-8c2e-f52416ba67f4" path="/var/lib/kubelet/pods/3ca05e63-f923-4eec-8c2e-f52416ba67f4/volumes" Oct 31 02:17:05.794031 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:05.793989 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:06.018688 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:06.018629 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:06.472372 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:06.472256 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:07.151927 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:07.150817 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:07.870796 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:07.870760 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:08.020383 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:08.020347 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:08.183810 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:08.181452 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7"] Oct 31 02:17:08.183810 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:08.181733 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" containerID="cri-o://224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633" gracePeriod=30 Oct 31 02:17:08.253247 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:08.253219 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:08.673740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:08.673223 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:09.551580 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:09.551516 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:10.259983 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:10.259948 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:10.972655 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:10.972618 5090 patch_prober.go:29] interesting pod/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:17:10.973069 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:10.972679 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:17:11.040464 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:11.040423 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:11.723429 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:11.723389 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:12.199065 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.199028 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677"] Oct 31 02:17:12.199567 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.199275 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" podUID="60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" containerName="maas-api" containerID="cri-o://2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74" gracePeriod=30 Oct 31 02:17:12.217790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.213889 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" podUID="60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.152:8443/health\": dial tcp 10.132.0.152:8443: connect: connection refused" Oct 31 02:17:12.301078 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.301046 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:12.483826 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.480606 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:12.583972 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.583943 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:17:12.623134 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.623077 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:12.647035 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.646996 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-kube-api-access-k7682\" (UniqueName: \"kubernetes.io/projected/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-kube-api-access-k7682\") pod \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " Oct 31 02:17:12.647239 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.647069 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-maas-api-tls\") pod \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " Oct 31 02:17:12.647239 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.647134 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-openshift-service-ca\") pod \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " Oct 31 02:17:12.647239 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.647196 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-metrics-certs\") pod \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\" (UID: \"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6\") " Oct 31 02:17:12.647584 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.647555 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-openshift-service-ca" pod "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" (UID: "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:12.647762 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.647682 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:12.649995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.649957 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-metrics-certs" pod "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" (UID: "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:12.650121 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.650022 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-maas-api-tls" pod "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" (UID: "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:12.650121 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.650058 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-kube-api-access-k7682" pod "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" (UID: "60c7a2e5-2f4c-405d-b81d-dcb75495ccd6"). InnerVolumeSpecName "kube-api-access-k7682". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:12.669264 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.669239 5090 generic.go:356] "Generic (PLEG): container finished" podID="60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" containerID="2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74" exitCode=0 Oct 31 02:17:12.669388 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.669291 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" Oct 31 02:17:12.669388 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.669328 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" event={"ID":"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6","Type":"ContainerDied","Data":"2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74"} Oct 31 02:17:12.669388 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.669382 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677" event={"ID":"60c7a2e5-2f4c-405d-b81d-dcb75495ccd6","Type":"ContainerDied","Data":"da330bee9a4ea2104ea4168bd638b065841dfc75ab91cf36b5536f04753d66ca"} Oct 31 02:17:12.669502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.669406 5090 scope.go:122] "RemoveContainer" containerID="2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74" Oct 31 02:17:12.680139 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.680119 5090 scope.go:122] "RemoveContainer" containerID="2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74" Oct 31 02:17:12.680455 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:12.680421 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74\": container with ID starting with 2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74 not found: ID does not exist" containerID="2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74" Oct 31 02:17:12.680508 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.680476 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74"} err="failed to get container status \"2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74\": rpc error: code = NotFound desc = could not find container \"2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74\": container with ID starting with 2aa370b1bdf8fe4d66bf6b4de814325693ad0a6374e66fc6611ccf252b86ee74 not found: ID does not exist" Oct 31 02:17:12.694336 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.694311 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677"] Oct 31 02:17:12.699434 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.699412 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d0c6ff-6ffbf74bc8-bp677"] Oct 31 02:17:12.748524 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.748493 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7682\" (UniqueName: \"kubernetes.io/projected/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-kube-api-access-k7682\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:12.748524 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.748523 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:12.748754 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.748534 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:12.757789 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:12.757757 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:13.039272 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.039188 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:13.459221 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.459199 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:17:13.556663 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556577 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-credential-socket\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.556663 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556653 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-certs\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.556939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556710 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-istio-token\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-istio-token\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.556939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556775 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-socket\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.556939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556812 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a52edf8-5178-419a-a452-19e019e5587a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a52edf8-5178-419a-a452-19e019e5587a-istio-podinfo\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.556939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556860 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-data\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.556939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556905 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a52edf8-5178-419a-a452-19e019e5587a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a52edf8-5178-419a-a452-19e019e5587a-istiod-ca-cert\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.556939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556919 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-credential-socket" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:17:13.556939 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.556944 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-envoy\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.557335 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.557009 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-kube-api-access-pjsj8\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-kube-api-access-pjsj8\") pod \"9a52edf8-5178-419a-a452-19e019e5587a\" (UID: \"9a52edf8-5178-419a-a452-19e019e5587a\") " Oct 31 02:17:13.557335 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.557323 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a52edf8-5178-419a-a452-19e019e5587a-istiod-ca-cert" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:13.557502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.557474 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-data" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:17:13.557635 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.557613 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a52edf8-5178-419a-a452-19e019e5587a-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.557635 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.557639 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.557871 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.557694 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-certs" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:17:13.557871 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.557821 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-socket" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:17:13.559865 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.559826 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-envoy" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:17:13.559970 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.559940 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-istio-token" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:13.560207 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.560188 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-kube-api-access-pjsj8" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "kube-api-access-pjsj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:13.561107 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.561080 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a52edf8-5178-419a-a452-19e019e5587a-istio-podinfo" pod "9a52edf8-5178-419a-a452-19e019e5587a" (UID: "9a52edf8-5178-419a-a452-19e019e5587a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:17:13.650059 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.650023 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:13.659324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.659287 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.659324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.659316 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.659324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.659326 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a52edf8-5178-419a-a452-19e019e5587a-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.659652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.659335 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.659652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.659344 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.659652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.659354 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjsj8\" (UniqueName: \"kubernetes.io/projected/9a52edf8-5178-419a-a452-19e019e5587a-kube-api-access-pjsj8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.659652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.659365 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a52edf8-5178-419a-a452-19e019e5587a-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:13.677364 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.677329 5090 generic.go:356] "Generic (PLEG): container finished" podID="9a52edf8-5178-419a-a452-19e019e5587a" containerID="224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633" exitCode=0 Oct 31 02:17:13.677528 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.677462 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" event={"ID":"9a52edf8-5178-419a-a452-19e019e5587a","Type":"ContainerDied","Data":"224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633"} Oct 31 02:17:13.677528 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.677505 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" event={"ID":"9a52edf8-5178-419a-a452-19e019e5587a","Type":"ContainerDied","Data":"d4b7148708c71bd85fa64543cc2867a0ec2bdcf7211055939d73611b5ee87d7a"} Oct 31 02:17:13.677643 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.677532 5090 scope.go:122] "RemoveContainer" containerID="224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633" Oct 31 02:17:13.677643 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.677464 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7" Oct 31 02:17:13.689614 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.689595 5090 scope.go:122] "RemoveContainer" containerID="224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633" Oct 31 02:17:13.689917 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:13.689898 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633\": container with ID starting with 224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633 not found: ID does not exist" containerID="224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633" Oct 31 02:17:13.689994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.689930 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633"} err="failed to get container status \"224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633\": rpc error: code = NotFound desc = could not find container \"224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633\": container with ID starting with 224ebb7adffe89c022bc7dbb33225e3446195f4e9709f90ab6c42ce137381633 not found: ID does not exist" Oct 31 02:17:13.703341 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.703316 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7"] Oct 31 02:17:13.709021 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.708994 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-bb5c51-openshift-default-7bfc99cf7c-z8md7"] Oct 31 02:17:13.747083 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.747052 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" path="/var/lib/kubelet/pods/60c7a2e5-2f4c-405d-b81d-dcb75495ccd6/volumes" Oct 31 02:17:13.747575 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:13.747554 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a52edf8-5178-419a-a452-19e019e5587a" path="/var/lib/kubelet/pods/9a52edf8-5178-419a-a452-19e019e5587a/volumes" Oct 31 02:17:14.330926 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:14.330891 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:15.023733 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:15.023649 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:15.666783 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:15.666252 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:15.982835 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:15.982758 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:16.137837 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:16.137803 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:16.335922 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:16.335834 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:16.944828 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:16.944793 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:17.638601 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:17.638553 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:18.159018 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:18.158979 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:18.296069 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:18.296032 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:18.441918 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:18.441823 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:18.585963 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:18.585930 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:18.894838 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:18.894804 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:19.038541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:19.038502 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:19.176606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:19.176523 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:19.312946 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:19.312908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:20.061526 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:20.061485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:20.351966 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:20.351886 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:20.495825 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:20.495795 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:20.735509 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:20.735362 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:20.877795 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:20.877707 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:21.556667 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:21.556546 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:22.262161 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:22.262086 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:22.777613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:22.777576 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:22.914388 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:22.914355 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:23.068512 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:23.068435 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:23.207391 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:23.207360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:23.597613 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:23.597583 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:23.894447 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:23.894363 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:24.648576 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:24.648536 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:25.434679 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:25.434649 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:25.889968 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:25.889936 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:26.028919 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:26.028885 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:26.196292 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:26.196198 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:26.333176 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:26.333139 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:26.838448 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:26.838374 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:27.570809 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:27.570772 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:28.170707 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:28.170597 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:28.825994 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:28.825954 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:29.583605 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:29.583568 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:30.196258 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:30.196227 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:30.330180 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:30.330146 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:30.687171 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:30.687135 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:31.051039 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:31.051004 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:31.449808 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:31.449674 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:31.600016 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:31.599975 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:31.765365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:31.765324 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt"] Oct 31 02:17:31.766290 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:31.766258 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" podUID="8afb6d7a-d18e-400c-b16a-0a7296ae1313" containerName="maas-api" containerID="cri-o://efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e" gracePeriod=30 Oct 31 02:17:31.833220 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:31.833191 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:32.026231 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.026005 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:32.113226 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.113198 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 02:17:32.183468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.183438 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:32.227215 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.227185 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-maas-api-tls\") pod \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " Oct 31 02:17:32.227388 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.227248 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afb6d7a-d18e-400c-b16a-0a7296ae1313-kube-api-access-9t9qd\" (UniqueName: \"kubernetes.io/projected/8afb6d7a-d18e-400c-b16a-0a7296ae1313-kube-api-access-9t9qd\") pod \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " Oct 31 02:17:32.227441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.227388 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-metrics-certs\") pod \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " Oct 31 02:17:32.227441 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.227427 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afb6d7a-d18e-400c-b16a-0a7296ae1313-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afb6d7a-d18e-400c-b16a-0a7296ae1313-openshift-service-ca\") pod \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\" (UID: \"8afb6d7a-d18e-400c-b16a-0a7296ae1313\") " Oct 31 02:17:32.227973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.227944 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afb6d7a-d18e-400c-b16a-0a7296ae1313-openshift-service-ca" pod "8afb6d7a-d18e-400c-b16a-0a7296ae1313" (UID: "8afb6d7a-d18e-400c-b16a-0a7296ae1313"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:32.229806 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.229769 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afb6d7a-d18e-400c-b16a-0a7296ae1313-kube-api-access-9t9qd" pod "8afb6d7a-d18e-400c-b16a-0a7296ae1313" (UID: "8afb6d7a-d18e-400c-b16a-0a7296ae1313"). InnerVolumeSpecName "kube-api-access-9t9qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:32.229806 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.229779 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-metrics-certs" pod "8afb6d7a-d18e-400c-b16a-0a7296ae1313" (UID: "8afb6d7a-d18e-400c-b16a-0a7296ae1313"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:32.230031 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.230003 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-maas-api-tls" pod "8afb6d7a-d18e-400c-b16a-0a7296ae1313" (UID: "8afb6d7a-d18e-400c-b16a-0a7296ae1313"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:32.328766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.328656 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:32.328766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.328687 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9qd\" (UniqueName: \"kubernetes.io/projected/8afb6d7a-d18e-400c-b16a-0a7296ae1313-kube-api-access-9t9qd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:32.328766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.328698 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8afb6d7a-d18e-400c-b16a-0a7296ae1313-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:32.328766 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.328745 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afb6d7a-d18e-400c-b16a-0a7296ae1313-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:32.329213 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.329030 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:32.726325 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.726241 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:32.802052 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.802018 5090 generic.go:356] "Generic (PLEG): container finished" podID="8afb6d7a-d18e-400c-b16a-0a7296ae1313" containerID="efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e" exitCode=0 Oct 31 02:17:32.802502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.802067 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" event={"ID":"8afb6d7a-d18e-400c-b16a-0a7296ae1313","Type":"ContainerDied","Data":"efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e"} Oct 31 02:17:32.802502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.802107 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" event={"ID":"8afb6d7a-d18e-400c-b16a-0a7296ae1313","Type":"ContainerDied","Data":"e6007fdc62ca8efb49a90746a52e3e19d1d09e0923b735f7052808863d9c0995"} Oct 31 02:17:32.802502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.802128 5090 scope.go:122] "RemoveContainer" containerID="efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e" Oct 31 02:17:32.802502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.802071 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt" Oct 31 02:17:32.816307 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.816282 5090 scope.go:122] "RemoveContainer" containerID="efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e" Oct 31 02:17:32.816640 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:32.816618 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e\": container with ID starting with efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e not found: ID does not exist" containerID="efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e" Oct 31 02:17:32.816755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.816649 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e"} err="failed to get container status \"efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e\": rpc error: code = NotFound desc = could not find container \"efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e\": container with ID starting with efa7b71e9da5d36f56d544aa5b13e6972ef8e95066dc65459366ea90d5b5724e not found: ID does not exist" Oct 31 02:17:32.828634 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.828603 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt"] Oct 31 02:17:32.832883 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.832859 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5555cb-58ccf8f944-gcmvt"] Oct 31 02:17:32.910001 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:32.909161 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:33.558136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:33.555322 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:33.746414 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:33.746377 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afb6d7a-d18e-400c-b16a-0a7296ae1313" path="/var/lib/kubelet/pods/8afb6d7a-d18e-400c-b16a-0a7296ae1313/volumes" Oct 31 02:17:34.241097 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:34.239363 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:34.788972 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:34.788931 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:17:34.929568 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:34.929477 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:35.520417 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:35.520386 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:36.253017 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:36.252947 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:36.955868 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:36.955802 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:37.139853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:37.139822 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:37.274287 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:37.274244 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:37.587420 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:37.587338 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:37.837745 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:37.837632 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:38.031090 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:38.031061 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:38.446083 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:38.446053 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:38.712371 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:38.712283 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:39.371443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:39.371405 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:39.921326 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:39.921276 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:40.056066 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:40.055802 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:40.287175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:40.287138 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:40.771910 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:40.771874 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:41.692268 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:41.692233 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:41.840548 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:41.840519 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:42.543950 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:42.543918 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:43.150681 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:43.150641 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:43.757669 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:43.757635 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:44.410393 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:44.410359 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:45.139628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.139544 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:45.398090 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.397998 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:45.545438 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.545405 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:45.612091 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.612042 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb"] Oct 31 02:17:45.612331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.612302 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" podUID="9a2b427a-d7ed-4b63-875e-d56ec348ccaf" containerName="maas-api" containerID="cri-o://cd498b8c14ae918279829abf12128737eabd78c27e57a11cd12f437d2719c31c" gracePeriod=30 Oct 31 02:17:45.720886 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.720846 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:45.864201 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.864167 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:45.903878 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.903848 5090 generic.go:356] "Generic (PLEG): container finished" podID="9a2b427a-d7ed-4b63-875e-d56ec348ccaf" containerID="cd498b8c14ae918279829abf12128737eabd78c27e57a11cd12f437d2719c31c" exitCode=0 Oct 31 02:17:45.904064 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.903921 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" event={"ID":"9a2b427a-d7ed-4b63-875e-d56ec348ccaf","Type":"ContainerDied","Data":"cd498b8c14ae918279829abf12128737eabd78c27e57a11cd12f437d2719c31c"} Oct 31 02:17:45.904064 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.903977 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" event={"ID":"9a2b427a-d7ed-4b63-875e-d56ec348ccaf","Type":"ContainerDied","Data":"35dc6cc5fefd7183dacd5c0e26c6ec2b95d71d0fe74f3b9ef360e642ca588467"} Oct 31 02:17:45.904064 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.903996 5090 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35dc6cc5fefd7183dacd5c0e26c6ec2b95d71d0fe74f3b9ef360e642ca588467" Oct 31 02:17:45.938859 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:45.937191 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 02:17:46.013443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.013405 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:46.098322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.098292 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-metrics-certs\") pod \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " Oct 31 02:17:46.098539 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.098350 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-openshift-service-ca\") pod \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " Oct 31 02:17:46.098539 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.098467 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-maas-api-tls\") pod \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " Oct 31 02:17:46.098539 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.098531 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-kube-api-access-tmbpn\" (UniqueName: \"kubernetes.io/projected/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-kube-api-access-tmbpn\") pod \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\" (UID: \"9a2b427a-d7ed-4b63-875e-d56ec348ccaf\") " Oct 31 02:17:46.098827 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.098797 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-openshift-service-ca" pod "9a2b427a-d7ed-4b63-875e-d56ec348ccaf" (UID: "9a2b427a-d7ed-4b63-875e-d56ec348ccaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:46.099019 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.098999 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:46.101251 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.101224 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-metrics-certs" pod "9a2b427a-d7ed-4b63-875e-d56ec348ccaf" (UID: "9a2b427a-d7ed-4b63-875e-d56ec348ccaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:46.101967 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.101936 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-maas-api-tls" pod "9a2b427a-d7ed-4b63-875e-d56ec348ccaf" (UID: "9a2b427a-d7ed-4b63-875e-d56ec348ccaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:46.102291 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.102267 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-kube-api-access-tmbpn" pod "9a2b427a-d7ed-4b63-875e-d56ec348ccaf" (UID: "9a2b427a-d7ed-4b63-875e-d56ec348ccaf"). InnerVolumeSpecName "kube-api-access-tmbpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:46.160816 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.160786 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:46.200094 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.200017 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:46.200094 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.200047 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmbpn\" (UniqueName: \"kubernetes.io/projected/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-kube-api-access-tmbpn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:46.200094 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.200057 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a2b427a-d7ed-4b63-875e-d56ec348ccaf-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:46.497973 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.497940 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:46.840583 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.840495 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:46.910442 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.910407 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb" Oct 31 02:17:46.934813 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.934775 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb"] Oct 31 02:17:46.940770 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:46.940736 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-ed4dce-54cd4966d7-7hqfb"] Oct 31 02:17:47.474464 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:47.474425 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:47.747235 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:47.747206 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a2b427a-d7ed-4b63-875e-d56ec348ccaf" path="/var/lib/kubelet/pods/9a2b427a-d7ed-4b63-875e-d56ec348ccaf/volumes" Oct 31 02:17:48.166463 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:48.166307 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:48.829295 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:48.829265 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:49.638228 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:49.637118 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:50.247373 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:50.247319 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:50.961051 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:50.961003 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:51.144935 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:51.144904 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:51.291708 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:51.291673 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:51.744959 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:51.744933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:52.345590 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:52.345563 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:52.490732 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:52.490686 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:52.986239 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:52.986203 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:53.133162 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.133125 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:53.367844 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.367813 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:53.492801 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.492767 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692"] Oct 31 02:17:53.493370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.493335 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" podUID="fc746fcd-8be9-4540-96fd-7e55187869be" containerName="maas-api" containerID="cri-o://7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1" gracePeriod=30 Oct 31 02:17:53.574030 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.573984 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:53.775465 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.775248 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:53.864626 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.864604 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 02:17:53.905763 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.905656 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:53.956303 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.956270 5090 generic.go:356] "Generic (PLEG): container finished" podID="fc746fcd-8be9-4540-96fd-7e55187869be" containerID="7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1" exitCode=0 Oct 31 02:17:53.956478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.956331 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" event={"ID":"fc746fcd-8be9-4540-96fd-7e55187869be","Type":"ContainerDied","Data":"7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1"} Oct 31 02:17:53.956478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.956369 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" event={"ID":"fc746fcd-8be9-4540-96fd-7e55187869be","Type":"ContainerDied","Data":"1e63fa25084faa662946bfc2675b7f58f261f76b0763520ba122967c89b6c8d4"} Oct 31 02:17:53.956478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.956392 5090 scope.go:122] "RemoveContainer" containerID="7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1" Oct 31 02:17:53.956478 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.956333 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692" Oct 31 02:17:53.966304 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.966286 5090 scope.go:122] "RemoveContainer" containerID="7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1" Oct 31 02:17:53.966607 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:53.966589 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1\": container with ID starting with 7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1 not found: ID does not exist" containerID="7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1" Oct 31 02:17:53.966658 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:53.966619 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1"} err="failed to get container status \"7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1\": rpc error: code = NotFound desc = could not find container \"7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1\": container with ID starting with 7d819f0acb74d4dc6d8f8050e8e76fee7136d9cde90080d3f475e6f765b9d7c1 not found: ID does not exist" Oct 31 02:17:54.011923 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.011882 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-metrics-certs\") pod \"fc746fcd-8be9-4540-96fd-7e55187869be\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " Oct 31 02:17:54.012110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.011942 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc746fcd-8be9-4540-96fd-7e55187869be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc746fcd-8be9-4540-96fd-7e55187869be-openshift-service-ca\") pod \"fc746fcd-8be9-4540-96fd-7e55187869be\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " Oct 31 02:17:54.012110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.011983 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls\") pod \"fc746fcd-8be9-4540-96fd-7e55187869be\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " Oct 31 02:17:54.012110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.012064 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc746fcd-8be9-4540-96fd-7e55187869be-kube-api-access-c9jbr\" (UniqueName: \"kubernetes.io/projected/fc746fcd-8be9-4540-96fd-7e55187869be-kube-api-access-c9jbr\") pod \"fc746fcd-8be9-4540-96fd-7e55187869be\" (UID: \"fc746fcd-8be9-4540-96fd-7e55187869be\") " Oct 31 02:17:54.013107 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.013075 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc746fcd-8be9-4540-96fd-7e55187869be-openshift-service-ca" pod "fc746fcd-8be9-4540-96fd-7e55187869be" (UID: "fc746fcd-8be9-4540-96fd-7e55187869be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:54.015660 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.015634 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc746fcd-8be9-4540-96fd-7e55187869be-kube-api-access-c9jbr" pod "fc746fcd-8be9-4540-96fd-7e55187869be" (UID: "fc746fcd-8be9-4540-96fd-7e55187869be"). InnerVolumeSpecName "kube-api-access-c9jbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:54.019147 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.019119 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-metrics-certs" pod "fc746fcd-8be9-4540-96fd-7e55187869be" (UID: "fc746fcd-8be9-4540-96fd-7e55187869be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:54.019260 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.019165 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls" pod "fc746fcd-8be9-4540-96fd-7e55187869be" (UID: "fc746fcd-8be9-4540-96fd-7e55187869be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:54.045361 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.045323 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:54.113728 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.113679 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:54.113728 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.113707 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc746fcd-8be9-4540-96fd-7e55187869be-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:54.113908 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.113736 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc746fcd-8be9-4540-96fd-7e55187869be-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:54.113908 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.113751 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jbr\" (UniqueName: \"kubernetes.io/projected/fc746fcd-8be9-4540-96fd-7e55187869be-kube-api-access-c9jbr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:54.279557 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.279527 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692"] Oct 31 02:17:54.286938 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.286910 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-89e955-776cb4b695-pz692"] Oct 31 02:17:54.424379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.424343 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:54.819217 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:54.819178 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:55.486430 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:55.485848 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:55.747570 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:55.747530 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc746fcd-8be9-4540-96fd-7e55187869be" path="/var/lib/kubelet/pods/fc746fcd-8be9-4540-96fd-7e55187869be/volumes" Oct 31 02:17:56.317109 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:56.317080 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:56.591838 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:56.591760 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:56.730398 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:56.730367 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:57.033534 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.033475 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:57.465602 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.465519 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c"] Oct 31 02:17:57.466070 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.466040 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" podUID="1a59d732-5ada-47c4-bc33-959413d7bd71" containerName="maas-api" containerID="cri-o://8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79" gracePeriod=30 Oct 31 02:17:57.554228 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.554182 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:57.696822 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.696629 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:57.833860 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.833831 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:57.835404 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.835383 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 02:17:57.889188 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.889145 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-maas-api-tls\") pod \"1a59d732-5ada-47c4-bc33-959413d7bd71\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " Oct 31 02:17:57.889395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.889286 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a59d732-5ada-47c4-bc33-959413d7bd71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a59d732-5ada-47c4-bc33-959413d7bd71-openshift-service-ca\") pod \"1a59d732-5ada-47c4-bc33-959413d7bd71\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " Oct 31 02:17:57.889395 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.889362 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-metrics-certs\") pod \"1a59d732-5ada-47c4-bc33-959413d7bd71\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " Oct 31 02:17:57.889623 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.889417 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a59d732-5ada-47c4-bc33-959413d7bd71-kube-api-access-4rwcq\" (UniqueName: \"kubernetes.io/projected/1a59d732-5ada-47c4-bc33-959413d7bd71-kube-api-access-4rwcq\") pod \"1a59d732-5ada-47c4-bc33-959413d7bd71\" (UID: \"1a59d732-5ada-47c4-bc33-959413d7bd71\") " Oct 31 02:17:57.890756 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.890687 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a59d732-5ada-47c4-bc33-959413d7bd71-openshift-service-ca" pod "1a59d732-5ada-47c4-bc33-959413d7bd71" (UID: "1a59d732-5ada-47c4-bc33-959413d7bd71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:17:57.893872 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.893842 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-metrics-certs" pod "1a59d732-5ada-47c4-bc33-959413d7bd71" (UID: "1a59d732-5ada-47c4-bc33-959413d7bd71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:57.894397 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.894356 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-maas-api-tls" pod "1a59d732-5ada-47c4-bc33-959413d7bd71" (UID: "1a59d732-5ada-47c4-bc33-959413d7bd71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:17:57.899033 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.898885 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a59d732-5ada-47c4-bc33-959413d7bd71-kube-api-access-4rwcq" pod "1a59d732-5ada-47c4-bc33-959413d7bd71" (UID: "1a59d732-5ada-47c4-bc33-959413d7bd71"). InnerVolumeSpecName "kube-api-access-4rwcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:17:57.972362 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.972131 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:57.991778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.991743 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:57.991778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.991772 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rwcq\" (UniqueName: \"kubernetes.io/projected/1a59d732-5ada-47c4-bc33-959413d7bd71-kube-api-access-4rwcq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:57.991982 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.991783 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a59d732-5ada-47c4-bc33-959413d7bd71-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:57.991982 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.991844 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a59d732-5ada-47c4-bc33-959413d7bd71-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:17:57.995367 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.995336 5090 generic.go:356] "Generic (PLEG): container finished" podID="1a59d732-5ada-47c4-bc33-959413d7bd71" containerID="8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79" exitCode=0 Oct 31 02:17:57.995508 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.995394 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" Oct 31 02:17:57.995508 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.995419 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" event={"ID":"1a59d732-5ada-47c4-bc33-959413d7bd71","Type":"ContainerDied","Data":"8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79"} Oct 31 02:17:57.995606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.995507 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c" event={"ID":"1a59d732-5ada-47c4-bc33-959413d7bd71","Type":"ContainerDied","Data":"f443a0956128be457dfd1e21a800ccc0e92ff38138ca12a0409ace93425b0e8d"} Oct 31 02:17:57.995606 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:57.995537 5090 scope.go:122] "RemoveContainer" containerID="8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79" Oct 31 02:17:58.008927 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:58.008904 5090 scope.go:122] "RemoveContainer" containerID="8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79" Oct 31 02:17:58.009198 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:17:58.009177 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79\": container with ID starting with 8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79 not found: ID does not exist" containerID="8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79" Oct 31 02:17:58.009319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:58.009209 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79"} err="failed to get container status \"8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79\": rpc error: code = NotFound desc = could not find container \"8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79\": container with ID starting with 8ef806a0a0a88bf3fc5faa855b21754ebcce60fa70393f4c0672d3eb669feb79 not found: ID does not exist" Oct 31 02:17:58.018582 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:58.018556 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c"] Oct 31 02:17:58.020906 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:58.020886 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-7a850e-f445985f5-d5d9c"] Oct 31 02:17:58.352443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:58.352389 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:58.632590 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:58.632497 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:59.346941 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:59.346899 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:17:59.747460 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:59.747431 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a59d732-5ada-47c4-bc33-959413d7bd71" path="/var/lib/kubelet/pods/1a59d732-5ada-47c4-bc33-959413d7bd71/volumes" Oct 31 02:17:59.955541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:17:59.955489 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:00.642088 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:00.642048 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:01.242848 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:01.242813 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:01.389149 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:01.389109 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:01.420210 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:01.420180 5090 scope.go:122] "RemoveContainer" containerID="cd498b8c14ae918279829abf12128737eabd78c27e57a11cd12f437d2719c31c" Oct 31 02:18:01.618948 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:01.618858 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:01.954054 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:01.953930 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:02.248790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:02.248749 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:02.957695 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:02.957663 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:03.751541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:03.751445 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:04.109458 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:04.109342 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:04.254159 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:04.254122 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:04.583210 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:04.583175 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:04.789954 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:18:04.789918 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:18:04.970732 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:04.970610 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:05.132083 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.132050 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f"] Oct 31 02:18:05.132553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.132347 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" podUID="eb6eb277-79de-4620-8243-4f41396ce223" containerName="maas-api" containerID="cri-o://ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7" gracePeriod=30 Oct 31 02:18:05.141585 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.141341 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:05.286286 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.286110 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:05.447976 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.436149 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:05.520032 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.519969 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 02:18:05.575413 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.575384 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:05.595315 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.595288 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-metrics-certs\") pod \"eb6eb277-79de-4620-8243-4f41396ce223\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " Oct 31 02:18:05.595467 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.595345 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb6eb277-79de-4620-8243-4f41396ce223-kube-api-access-2p54v\" (UniqueName: \"kubernetes.io/projected/eb6eb277-79de-4620-8243-4f41396ce223-kube-api-access-2p54v\") pod \"eb6eb277-79de-4620-8243-4f41396ce223\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " Oct 31 02:18:05.595467 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.595423 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls\") pod \"eb6eb277-79de-4620-8243-4f41396ce223\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " Oct 31 02:18:05.595554 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.595465 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb6eb277-79de-4620-8243-4f41396ce223-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6eb277-79de-4620-8243-4f41396ce223-openshift-service-ca\") pod \"eb6eb277-79de-4620-8243-4f41396ce223\" (UID: \"eb6eb277-79de-4620-8243-4f41396ce223\") " Oct 31 02:18:05.596146 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.596115 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb6eb277-79de-4620-8243-4f41396ce223-openshift-service-ca" pod "eb6eb277-79de-4620-8243-4f41396ce223" (UID: "eb6eb277-79de-4620-8243-4f41396ce223"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:18:05.598177 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.598149 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls" pod "eb6eb277-79de-4620-8243-4f41396ce223" (UID: "eb6eb277-79de-4620-8243-4f41396ce223"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:05.598299 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.598183 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-metrics-certs" pod "eb6eb277-79de-4620-8243-4f41396ce223" (UID: "eb6eb277-79de-4620-8243-4f41396ce223"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:05.598572 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.598546 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb6eb277-79de-4620-8243-4f41396ce223-kube-api-access-2p54v" pod "eb6eb277-79de-4620-8243-4f41396ce223" (UID: "eb6eb277-79de-4620-8243-4f41396ce223"). InnerVolumeSpecName "kube-api-access-2p54v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:05.696594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.696555 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:05.696594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.696590 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb6eb277-79de-4620-8243-4f41396ce223-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:05.696594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.696601 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb6eb277-79de-4620-8243-4f41396ce223-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:05.696879 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.696610 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p54v\" (UniqueName: \"kubernetes.io/projected/eb6eb277-79de-4620-8243-4f41396ce223-kube-api-access-2p54v\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:05.716675 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:05.716611 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:06.045459 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.045425 5090 generic.go:356] "Generic (PLEG): container finished" podID="eb6eb277-79de-4620-8243-4f41396ce223" containerID="ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7" exitCode=0 Oct 31 02:18:06.045645 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.045466 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" Oct 31 02:18:06.045645 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.045487 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" event={"ID":"eb6eb277-79de-4620-8243-4f41396ce223","Type":"ContainerDied","Data":"ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7"} Oct 31 02:18:06.045645 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.045529 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f" event={"ID":"eb6eb277-79de-4620-8243-4f41396ce223","Type":"ContainerDied","Data":"60aa4159b4395a02e733f4ac0f1c274fcd993c720d26e85d5aea397a0c9f845b"} Oct 31 02:18:06.045645 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.045549 5090 scope.go:122] "RemoveContainer" containerID="ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7" Oct 31 02:18:06.048063 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.048034 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:06.055808 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.055790 5090 scope.go:122] "RemoveContainer" containerID="ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7" Oct 31 02:18:06.056079 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:18:06.056062 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7\": container with ID starting with ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7 not found: ID does not exist" containerID="ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7" Oct 31 02:18:06.056127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.056089 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7"} err="failed to get container status \"ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7\": rpc error: code = NotFound desc = could not find container \"ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7\": container with ID starting with ecbc96e1701d74ba47fc77f882323821cbbe64e2174aac69720fed6537a32cc7 not found: ID does not exist" Oct 31 02:18:06.062558 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.062539 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f"] Oct 31 02:18:06.070026 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.069999 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2e6bfd-b8f6d648d-dxq5f"] Oct 31 02:18:06.457795 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:06.456362 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:07.151861 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:07.151833 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:07.291421 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:07.291390 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:07.748317 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:07.748278 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb6eb277-79de-4620-8243-4f41396ce223" path="/var/lib/kubelet/pods/eb6eb277-79de-4620-8243-4f41396ce223/volumes" Oct 31 02:18:07.894771 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:07.894734 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:08.643627 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:08.643578 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:09.360168 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:09.360127 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:10.056243 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:10.056173 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:10.863925 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:10.863891 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:11.436305 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:11.436267 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:12.142866 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:12.142378 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:12.553647 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:12.553617 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:12.690955 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:12.690917 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:12.828616 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:12.828535 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:12.968999 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:12.968955 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:13.193823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:13.193673 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:13.559737 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:13.557913 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:13.696243 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:13.696214 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:13.832823 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:13.832745 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:14.453878 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:14.453844 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:15.310752 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:15.307525 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:15.916653 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:15.916615 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq"] Oct 31 02:18:16.037385 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:16.037353 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:16.177333 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:16.177246 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:16.320486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:16.320455 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:16.994394 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:16.994360 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:17.652643 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:17.652601 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:18.064370 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.064334 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt"] Oct 31 02:18:18.064880 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.064588 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" podUID="f3b2b839-b6a1-46fe-8678-c6d984684533" containerName="maas-api" containerID="cri-o://0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526" gracePeriod=30 Oct 31 02:18:18.158652 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.158616 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:18.320145 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.320054 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:18.428495 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.428472 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 02:18:18.488914 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.488879 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:18.493288 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.493255 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b2b839-b6a1-46fe-8678-c6d984684533-kube-api-access-cwgzh\" (UniqueName: \"kubernetes.io/projected/f3b2b839-b6a1-46fe-8678-c6d984684533-kube-api-access-cwgzh\") pod \"f3b2b839-b6a1-46fe-8678-c6d984684533\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " Oct 31 02:18:18.493409 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.493360 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b2b839-b6a1-46fe-8678-c6d984684533-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2b839-b6a1-46fe-8678-c6d984684533-openshift-service-ca\") pod \"f3b2b839-b6a1-46fe-8678-c6d984684533\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " Oct 31 02:18:18.493482 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.493433 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-metrics-certs\") pod \"f3b2b839-b6a1-46fe-8678-c6d984684533\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " Oct 31 02:18:18.493482 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.493469 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-maas-api-tls\") pod \"f3b2b839-b6a1-46fe-8678-c6d984684533\" (UID: \"f3b2b839-b6a1-46fe-8678-c6d984684533\") " Oct 31 02:18:18.493794 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.493769 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b2b839-b6a1-46fe-8678-c6d984684533-openshift-service-ca" pod "f3b2b839-b6a1-46fe-8678-c6d984684533" (UID: "f3b2b839-b6a1-46fe-8678-c6d984684533"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:18:18.493946 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.493916 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3b2b839-b6a1-46fe-8678-c6d984684533-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:18.495870 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.495842 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-metrics-certs" pod "f3b2b839-b6a1-46fe-8678-c6d984684533" (UID: "f3b2b839-b6a1-46fe-8678-c6d984684533"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:18.495976 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.495866 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-maas-api-tls" pod "f3b2b839-b6a1-46fe-8678-c6d984684533" (UID: "f3b2b839-b6a1-46fe-8678-c6d984684533"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:18.496066 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.496041 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b2b839-b6a1-46fe-8678-c6d984684533-kube-api-access-cwgzh" pod "f3b2b839-b6a1-46fe-8678-c6d984684533" (UID: "f3b2b839-b6a1-46fe-8678-c6d984684533"). InnerVolumeSpecName "kube-api-access-cwgzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:18.595534 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.595455 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwgzh\" (UniqueName: \"kubernetes.io/projected/f3b2b839-b6a1-46fe-8678-c6d984684533-kube-api-access-cwgzh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:18.595534 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.595487 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:18.595534 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.595497 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3b2b839-b6a1-46fe-8678-c6d984684533-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:18.688541 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.688511 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:18.804155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.804122 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr"] Oct 31 02:18:18.950111 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:18.950008 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:19.088110 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.088060 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:19.126151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.126117 5090 generic.go:356] "Generic (PLEG): container finished" podID="f3b2b839-b6a1-46fe-8678-c6d984684533" containerID="0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526" exitCode=0 Oct 31 02:18:19.126151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.126154 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" event={"ID":"f3b2b839-b6a1-46fe-8678-c6d984684533","Type":"ContainerDied","Data":"0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526"} Oct 31 02:18:19.126351 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.126200 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" event={"ID":"f3b2b839-b6a1-46fe-8678-c6d984684533","Type":"ContainerDied","Data":"47150cba7581bbfa6d819ccb1f2c3bad3c27f7d3be56eae551bd38d677089c92"} Oct 31 02:18:19.126351 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.126200 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt" Oct 31 02:18:19.126351 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.126216 5090 scope.go:122] "RemoveContainer" containerID="0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526" Oct 31 02:18:19.137117 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.137095 5090 scope.go:122] "RemoveContainer" containerID="0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526" Oct 31 02:18:19.137377 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:18:19.137351 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526\": container with ID starting with 0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526 not found: ID does not exist" containerID="0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526" Oct 31 02:18:19.137460 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.137390 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526"} err="failed to get container status \"0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526\": rpc error: code = NotFound desc = could not find container \"0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526\": container with ID starting with 0b433095275e91d45e6126118bbbe319da8fe15e153cdf33c25cd6346a405526 not found: ID does not exist" Oct 31 02:18:19.150301 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.150252 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt"] Oct 31 02:18:19.153153 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.153107 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae5a1c-76cfb55d9d-97ctt"] Oct 31 02:18:19.370963 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.370927 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:19.746864 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:19.746827 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b2b839-b6a1-46fe-8678-c6d984684533" path="/var/lib/kubelet/pods/f3b2b839-b6a1-46fe-8678-c6d984684533/volumes" Oct 31 02:18:20.264220 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:20.263433 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:20.916988 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:20.916949 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" podUID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" containerName="payload-processing" containerID="cri-o://f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9" gracePeriod=25 Oct 31 02:18:20.970863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:20.970814 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:21.769626 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:21.769588 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:22.575365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:22.575332 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:23.343966 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:23.343933 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:24.057532 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:24.057485 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:24.207266 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:24.207230 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:24.340532 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:24.340458 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:24.919016 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:24.918981 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:25.536506 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:25.536476 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:25.675675 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:25.675643 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:25.905908 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:25.905830 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:26.252493 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:26.252464 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:26.419791 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:26.419760 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:26.573746 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:26.573632 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:26.717486 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:26.717449 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:27.023475 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:27.023444 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:27.764896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:27.764860 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:27.998925 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:27.998888 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" podUID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:18:28.461561 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:28.461526 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:29.114628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:29.114591 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:29.849569 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:29.849533 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:30.559005 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:30.558961 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:31.257678 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:31.257644 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:31.522136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:31.521853 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:32.036427 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:32.036388 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:32.767041 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:32.766968 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:33.446756 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:33.446648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:33.826291 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:33.826251 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" podUID="5f19ad8a-396e-4322-abf4-f88698c9893c" containerName="main" containerID="cri-o://e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd" gracePeriod=45 Oct 31 02:18:34.086499 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.086438 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 02:18:34.154975 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.154932 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-tmp-dir\") pod \"5f19ad8a-396e-4322-abf4-f88698c9893c\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " Oct 31 02:18:34.155171 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155013 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f19ad8a-396e-4322-abf4-f88698c9893c-kube-api-access-7j4zp\" (UniqueName: \"kubernetes.io/projected/5f19ad8a-396e-4322-abf4-f88698c9893c-kube-api-access-7j4zp\") pod \"5f19ad8a-396e-4322-abf4-f88698c9893c\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " Oct 31 02:18:34.155171 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155071 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f19ad8a-396e-4322-abf4-f88698c9893c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f19ad8a-396e-4322-abf4-f88698c9893c-tls-certs\") pod \"5f19ad8a-396e-4322-abf4-f88698c9893c\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " Oct 31 02:18:34.155171 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155119 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-model-cache\") pod \"5f19ad8a-396e-4322-abf4-f88698c9893c\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " Oct 31 02:18:34.155366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155221 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-tmp-dir" pod "5f19ad8a-396e-4322-abf4-f88698c9893c" (UID: "5f19ad8a-396e-4322-abf4-f88698c9893c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:34.155366 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155257 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-home\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-home\") pod \"5f19ad8a-396e-4322-abf4-f88698c9893c\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " Oct 31 02:18:34.155525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155502 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-home" pod "5f19ad8a-396e-4322-abf4-f88698c9893c" (UID: "5f19ad8a-396e-4322-abf4-f88698c9893c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:34.155793 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155760 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-model-cache" pod "5f19ad8a-396e-4322-abf4-f88698c9893c" (UID: "5f19ad8a-396e-4322-abf4-f88698c9893c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:34.155990 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.155778 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-dshm\") pod \"5f19ad8a-396e-4322-abf4-f88698c9893c\" (UID: \"5f19ad8a-396e-4322-abf4-f88698c9893c\") " Oct 31 02:18:34.156529 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.156507 5090 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:34.156668 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.156651 5090 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:34.156867 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.156851 5090 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:34.157997 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.157972 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f19ad8a-396e-4322-abf4-f88698c9893c-tls-certs" pod "5f19ad8a-396e-4322-abf4-f88698c9893c" (UID: "5f19ad8a-396e-4322-abf4-f88698c9893c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:34.158628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.158584 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f19ad8a-396e-4322-abf4-f88698c9893c-kube-api-access-7j4zp" pod "5f19ad8a-396e-4322-abf4-f88698c9893c" (UID: "5f19ad8a-396e-4322-abf4-f88698c9893c"). InnerVolumeSpecName "kube-api-access-7j4zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:34.158628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.158590 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-dshm" pod "5f19ad8a-396e-4322-abf4-f88698c9893c" (UID: "5f19ad8a-396e-4322-abf4-f88698c9893c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:34.221466 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.221426 5090 generic.go:356] "Generic (PLEG): container finished" podID="5f19ad8a-396e-4322-abf4-f88698c9893c" containerID="e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd" exitCode=0 Oct 31 02:18:34.221466 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.221472 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" Oct 31 02:18:34.221750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.221589 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" event={"ID":"5f19ad8a-396e-4322-abf4-f88698c9893c","Type":"ContainerDied","Data":"e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd"} Oct 31 02:18:34.221750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.221620 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr" event={"ID":"5f19ad8a-396e-4322-abf4-f88698c9893c","Type":"ContainerDied","Data":"008f68d7425ce0dcc082d73c86980f79008ff4559a477c4888f469a014835e70"} Oct 31 02:18:34.221750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.221639 5090 scope.go:122] "RemoveContainer" containerID="e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd" Oct 31 02:18:34.237418 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.237394 5090 scope.go:122] "RemoveContainer" containerID="e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd" Oct 31 02:18:34.237750 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:18:34.237709 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd\": container with ID starting with e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd not found: ID does not exist" containerID="e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd" Oct 31 02:18:34.237837 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.237763 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd"} err="failed to get container status \"e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd\": rpc error: code = NotFound desc = could not find container \"e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd\": container with ID starting with e4a0c696ecdaedfd0cb4b0c25ee4b169e10facc2b0302311f4a8adda0bda1fdd not found: ID does not exist" Oct 31 02:18:34.249072 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.249014 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr"] Oct 31 02:18:34.252770 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.252743 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6ae519/premium-w2-6ae519-kserve-847fd96895-h2rgr"] Oct 31 02:18:34.258381 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.258357 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4zp\" (UniqueName: \"kubernetes.io/projected/5f19ad8a-396e-4322-abf4-f88698c9893c-kube-api-access-7j4zp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:34.258496 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.258385 5090 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f19ad8a-396e-4322-abf4-f88698c9893c-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:34.258496 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.258400 5090 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f19ad8a-396e-4322-abf4-f88698c9893c-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:34.422991 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.422908 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:34.739503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.739424 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:34.805768 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:18:34.805710 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:18:34.879690 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:34.879657 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:35.020628 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:35.020592 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt"] Oct 31 02:18:35.021146 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:35.021110 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" containerID="cri-o://7b6fd1a7a5d30376549b5cc9c14b3319baeb617ebf4c767f1945c94d2930548d" gracePeriod=30 Oct 31 02:18:35.090499 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:35.090453 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:35.361657 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:35.361572 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:35.745949 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:35.745923 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f19ad8a-396e-4322-abf4-f88698c9893c" path="/var/lib/kubelet/pods/5f19ad8a-396e-4322-abf4-f88698c9893c/volumes" Oct 31 02:18:35.944075 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:35.944036 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:36.623080 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:36.623049 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:36.907960 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:36.907888 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:37.066462 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:37.066420 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:37.998185 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:37.998151 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" podUID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 02:18:39.124087 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.124044 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x"] Oct 31 02:18:39.124578 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.124324 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" podUID="eef7beaa-db5f-408c-bd7a-b74a74248d63" containerName="maas-api" containerID="cri-o://850f8a536e1d17e7aa1ab32424d015ed57abd2204fd62ea2bb8e4e308c506bd5" gracePeriod=30 Oct 31 02:18:39.200682 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.200648 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:39.263790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.263752 5090 generic.go:356] "Generic (PLEG): container finished" podID="eef7beaa-db5f-408c-bd7a-b74a74248d63" containerID="850f8a536e1d17e7aa1ab32424d015ed57abd2204fd62ea2bb8e4e308c506bd5" exitCode=0 Oct 31 02:18:39.263790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.263772 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" event={"ID":"eef7beaa-db5f-408c-bd7a-b74a74248d63","Type":"ContainerDied","Data":"850f8a536e1d17e7aa1ab32424d015ed57abd2204fd62ea2bb8e4e308c506bd5"} Oct 31 02:18:39.375806 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.373525 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:39.501730 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.501689 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 02:18:39.513142 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.513103 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:39.651175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.651084 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eef7beaa-db5f-408c-bd7a-b74a74248d63-kube-api-access-q2mwg\" (UniqueName: \"kubernetes.io/projected/eef7beaa-db5f-408c-bd7a-b74a74248d63-kube-api-access-q2mwg\") pod \"eef7beaa-db5f-408c-bd7a-b74a74248d63\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " Oct 31 02:18:39.651175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.651138 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eef7beaa-db5f-408c-bd7a-b74a74248d63-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef7beaa-db5f-408c-bd7a-b74a74248d63-openshift-service-ca\") pod \"eef7beaa-db5f-408c-bd7a-b74a74248d63\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " Oct 31 02:18:39.651175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.651174 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-maas-api-tls\") pod \"eef7beaa-db5f-408c-bd7a-b74a74248d63\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " Oct 31 02:18:39.651503 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.651292 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-metrics-certs\") pod \"eef7beaa-db5f-408c-bd7a-b74a74248d63\" (UID: \"eef7beaa-db5f-408c-bd7a-b74a74248d63\") " Oct 31 02:18:39.652011 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.651902 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef7beaa-db5f-408c-bd7a-b74a74248d63-openshift-service-ca" pod "eef7beaa-db5f-408c-bd7a-b74a74248d63" (UID: "eef7beaa-db5f-408c-bd7a-b74a74248d63"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:18:39.653846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.653820 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-metrics-certs" pod "eef7beaa-db5f-408c-bd7a-b74a74248d63" (UID: "eef7beaa-db5f-408c-bd7a-b74a74248d63"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:39.654049 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.653983 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef7beaa-db5f-408c-bd7a-b74a74248d63-kube-api-access-q2mwg" pod "eef7beaa-db5f-408c-bd7a-b74a74248d63" (UID: "eef7beaa-db5f-408c-bd7a-b74a74248d63"). InnerVolumeSpecName "kube-api-access-q2mwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:39.655080 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.655049 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-maas-api-tls" pod "eef7beaa-db5f-408c-bd7a-b74a74248d63" (UID: "eef7beaa-db5f-408c-bd7a-b74a74248d63"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:39.752322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.752289 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:39.752322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.752317 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eef7beaa-db5f-408c-bd7a-b74a74248d63-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:39.752322 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.752328 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2mwg\" (UniqueName: \"kubernetes.io/projected/eef7beaa-db5f-408c-bd7a-b74a74248d63-kube-api-access-q2mwg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:39.752563 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.752338 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef7beaa-db5f-408c-bd7a-b74a74248d63-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:39.901811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:39.901695 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:40.271668 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.271642 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" Oct 31 02:18:40.272153 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.271659 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x" event={"ID":"eef7beaa-db5f-408c-bd7a-b74a74248d63","Type":"ContainerDied","Data":"24f7924d50ebdf48d815a7575fc48b0636b265d6790ecbd43aba121c3368168c"} Oct 31 02:18:40.272153 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.271751 5090 scope.go:122] "RemoveContainer" containerID="850f8a536e1d17e7aa1ab32424d015ed57abd2204fd62ea2bb8e4e308c506bd5" Oct 31 02:18:40.273121 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.273101 5090 generic.go:356] "Generic (PLEG): container finished" podID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerID="7b6fd1a7a5d30376549b5cc9c14b3319baeb617ebf4c767f1945c94d2930548d" exitCode=0 Oct 31 02:18:40.273219 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.273175 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" event={"ID":"b17cbd44-ea63-47ff-8af6-689574aaea89","Type":"ContainerDied","Data":"7b6fd1a7a5d30376549b5cc9c14b3319baeb617ebf4c767f1945c94d2930548d"} Oct 31 02:18:40.297227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.297206 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 02:18:40.308122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.308097 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x"] Oct 31 02:18:40.314355 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.314331 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-6ae519-7dd6567654-xx67x"] Oct 31 02:18:40.358353 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358319 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-credential-socket\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358495 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358381 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-envoy\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358495 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358484 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-data\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358598 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358511 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17cbd44-ea63-47ff-8af6-689574aaea89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17cbd44-ea63-47ff-8af6-689574aaea89-istiod-ca-cert\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358598 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358578 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-podinfo\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358748 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358637 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-certs\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358748 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358642 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-credential-socket" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:40.358875 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358775 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-socket\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358875 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358828 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-kube-api-access-gcpv4\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-kube-api-access-gcpv4\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358875 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358858 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-token\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-token\") pod \"b17cbd44-ea63-47ff-8af6-689574aaea89\" (UID: \"b17cbd44-ea63-47ff-8af6-689574aaea89\") " Oct 31 02:18:40.358875 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.358861 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-certs" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:40.359203 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.359066 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-socket" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:40.359317 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.359296 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.359386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.359180 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17cbd44-ea63-47ff-8af6-689574aaea89-istiod-ca-cert" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:18:40.359386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.359324 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.359386 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.359344 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.359566 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.359475 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-data" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:40.361357 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.361331 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-token" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:40.361456 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.361402 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-envoy" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:40.361611 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.361585 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-kube-api-access-gcpv4" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "kube-api-access-gcpv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:40.361740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.361682 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-podinfo" pod "b17cbd44-ea63-47ff-8af6-689574aaea89" (UID: "b17cbd44-ea63-47ff-8af6-689574aaea89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:18:40.460209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.460138 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.460209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.460166 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17cbd44-ea63-47ff-8af6-689574aaea89-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.460209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.460177 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.460209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.460185 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcpv4\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-kube-api-access-gcpv4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.460209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.460196 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:40.460209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:40.460204 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17cbd44-ea63-47ff-8af6-689574aaea89-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:41.169047 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.169022 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 02:18:41.269034 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.269002 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-kube-api-access-nbcnk\" (UniqueName: \"kubernetes.io/projected/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-kube-api-access-nbcnk\") pod \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " Oct 31 02:18:41.269227 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.269146 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-plugins-config-volume\") pod \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\" (UID: \"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7\") " Oct 31 02:18:41.269533 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.269508 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-plugins-config-volume" pod "abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" (UID: "abf2337f-bdd6-4ed5-ad43-83c528eb8cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:18:41.271289 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.271267 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-kube-api-access-nbcnk" pod "abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" (UID: "abf2337f-bdd6-4ed5-ad43-83c528eb8cb7"). InnerVolumeSpecName "kube-api-access-nbcnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:41.280447 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.280423 5090 generic.go:356] "Generic (PLEG): container finished" podID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" containerID="f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9" exitCode=0 Oct 31 02:18:41.280811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.280471 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" Oct 31 02:18:41.280811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.280509 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" event={"ID":"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7","Type":"ContainerDied","Data":"f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9"} Oct 31 02:18:41.280811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.280562 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq" event={"ID":"abf2337f-bdd6-4ed5-ad43-83c528eb8cb7","Type":"ContainerDied","Data":"f12094e36f0b1af91205e9d1cd61c594613201384ca060474f741d8e8b0c8efd"} Oct 31 02:18:41.280811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.280580 5090 scope.go:122] "RemoveContainer" containerID="f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9" Oct 31 02:18:41.282439 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.282197 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" event={"ID":"b17cbd44-ea63-47ff-8af6-689574aaea89","Type":"ContainerDied","Data":"92df8b90e0bdcdf5be4c54ceacb9399abdc63b9478212efc6faa505366810127"} Oct 31 02:18:41.282439 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.282212 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt" Oct 31 02:18:41.290365 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.290345 5090 scope.go:122] "RemoveContainer" containerID="f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9" Oct 31 02:18:41.290643 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:18:41.290623 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9\": container with ID starting with f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9 not found: ID does not exist" containerID="f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9" Oct 31 02:18:41.290779 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.290650 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9"} err="failed to get container status \"f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9\": rpc error: code = NotFound desc = could not find container \"f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9\": container with ID starting with f913993a5f6a71032771a932a5a93e2e0d0d5ab8717fb16e2b07c9fe0411e8c9 not found: ID does not exist" Oct 31 02:18:41.290779 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.290666 5090 scope.go:122] "RemoveContainer" containerID="7b6fd1a7a5d30376549b5cc9c14b3319baeb617ebf4c767f1945c94d2930548d" Oct 31 02:18:41.305567 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.305547 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq"] Oct 31 02:18:41.310287 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.310267 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae5a1c-9d8d8c57c-9zssq"] Oct 31 02:18:41.321215 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.321194 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt"] Oct 31 02:18:41.325964 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.325936 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0e8b5c-openshift-default-6786f5c45-qgldt"] Oct 31 02:18:41.370601 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.370579 5090 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:41.370601 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.370601 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbcnk\" (UniqueName: \"kubernetes.io/projected/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7-kube-api-access-nbcnk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:41.745553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.745522 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" path="/var/lib/kubelet/pods/abf2337f-bdd6-4ed5-ad43-83c528eb8cb7/volumes" Oct 31 02:18:41.745995 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.745980 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" path="/var/lib/kubelet/pods/b17cbd44-ea63-47ff-8af6-689574aaea89/volumes" Oct 31 02:18:41.746359 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:41.746345 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eef7beaa-db5f-408c-bd7a-b74a74248d63" path="/var/lib/kubelet/pods/eef7beaa-db5f-408c-bd7a-b74a74248d63/volumes" Oct 31 02:18:45.364846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:45.364809 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:45.497839 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:45.497800 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:45.865535 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:45.865499 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:46.910948 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:46.910900 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:47.050588 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:47.050555 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:47.193809 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:47.186758 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l"] Oct 31 02:18:47.193809 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:47.186997 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" containerID="cri-o://97d8dfc88c65f951a5797f36b4b416e46828a9c1cd12b1ee85f20d58ec8e1e25" gracePeriod=30 Oct 31 02:18:47.255354 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:47.255313 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:49.689817 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:49.689428 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6"] Oct 31 02:18:49.690379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:49.689918 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" podUID="b3a029da-d748-484f-a127-645719398a9c" containerName="maas-api" containerID="cri-o://057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585" gracePeriod=30 Oct 31 02:18:49.775772 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:49.775741 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:49.928127 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:49.928090 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:50.006703 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.006672 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 02:18:50.071291 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.071254 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:50.166710 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.166675 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3a029da-d748-484f-a127-645719398a9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a029da-d748-484f-a127-645719398a9c-openshift-service-ca\") pod \"b3a029da-d748-484f-a127-645719398a9c\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " Oct 31 02:18:50.166887 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.166857 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-metrics-certs\") pod \"b3a029da-d748-484f-a127-645719398a9c\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " Oct 31 02:18:50.166944 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.166933 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3a029da-d748-484f-a127-645719398a9c-kube-api-access-cd6r4\" (UniqueName: \"kubernetes.io/projected/b3a029da-d748-484f-a127-645719398a9c-kube-api-access-cd6r4\") pod \"b3a029da-d748-484f-a127-645719398a9c\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " Oct 31 02:18:50.167036 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.166974 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-maas-api-tls\") pod \"b3a029da-d748-484f-a127-645719398a9c\" (UID: \"b3a029da-d748-484f-a127-645719398a9c\") " Oct 31 02:18:50.167098 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.167077 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a029da-d748-484f-a127-645719398a9c-openshift-service-ca" pod "b3a029da-d748-484f-a127-645719398a9c" (UID: "b3a029da-d748-484f-a127-645719398a9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:18:50.167323 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.167298 5090 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a029da-d748-484f-a127-645719398a9c-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:50.169331 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.169301 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-metrics-certs" pod "b3a029da-d748-484f-a127-645719398a9c" (UID: "b3a029da-d748-484f-a127-645719398a9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:50.169896 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.169866 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a029da-d748-484f-a127-645719398a9c-kube-api-access-cd6r4" pod "b3a029da-d748-484f-a127-645719398a9c" (UID: "b3a029da-d748-484f-a127-645719398a9c"). InnerVolumeSpecName "kube-api-access-cd6r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:50.170003 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.169914 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-maas-api-tls" pod "b3a029da-d748-484f-a127-645719398a9c" (UID: "b3a029da-d748-484f-a127-645719398a9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 02:18:50.268182 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.268152 5090 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:50.268182 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.268183 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6r4\" (UniqueName: \"kubernetes.io/projected/b3a029da-d748-484f-a127-645719398a9c-kube-api-access-cd6r4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:50.268373 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.268194 5090 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3a029da-d748-484f-a127-645719398a9c-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:50.347357 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.347313 5090 generic.go:356] "Generic (PLEG): container finished" podID="b3a029da-d748-484f-a127-645719398a9c" containerID="057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585" exitCode=0 Oct 31 02:18:50.347519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.347365 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" Oct 31 02:18:50.347519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.347396 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" event={"ID":"b3a029da-d748-484f-a127-645719398a9c","Type":"ContainerDied","Data":"057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585"} Oct 31 02:18:50.347519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.347451 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6" event={"ID":"b3a029da-d748-484f-a127-645719398a9c","Type":"ContainerDied","Data":"809f0348f279217f3a2b8b218dc0361f4234894fd43e7055df16919ae7ea875d"} Oct 31 02:18:50.347519 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.347494 5090 scope.go:122] "RemoveContainer" containerID="057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585" Oct 31 02:18:50.357566 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.357548 5090 scope.go:122] "RemoveContainer" containerID="057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585" Oct 31 02:18:50.357912 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:18:50.357884 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585\": container with ID starting with 057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585 not found: ID does not exist" containerID="057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585" Oct 31 02:18:50.357975 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.357924 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585"} err="failed to get container status \"057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585\": rpc error: code = NotFound desc = could not find container \"057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585\": container with ID starting with 057a7b470352fe6cd912c039f339aa8fbaa3cb655202709f0b1adb0d599cf585 not found: ID does not exist" Oct 31 02:18:50.368755 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.368732 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6"] Oct 31 02:18:50.392597 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.392574 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7e5955-fb6df4546-9tll6"] Oct 31 02:18:50.618741 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.618646 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:50.631485 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.631448 5090 patch_prober.go:29] interesting pod/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:18:50.631620 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:50.631515 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:18:51.753155 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:51.753120 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3a029da-d748-484f-a127-645719398a9c" path="/var/lib/kubelet/pods/b3a029da-d748-484f-a127-645719398a9c/volumes" Oct 31 02:18:51.988244 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:51.988208 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:52.361848 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.361814 5090 generic.go:356] "Generic (PLEG): container finished" podID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerID="97d8dfc88c65f951a5797f36b4b416e46828a9c1cd12b1ee85f20d58ec8e1e25" exitCode=0 Oct 31 02:18:52.361992 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.361886 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" event={"ID":"bb7bc789-200b-468c-8525-5b67443e9ed9","Type":"ContainerDied","Data":"97d8dfc88c65f951a5797f36b4b416e46828a9c1cd12b1ee85f20d58ec8e1e25"} Oct 31 02:18:52.454250 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.454222 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:52.459985 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.459924 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 02:18:52.591993 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.591961 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-envoy\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592199 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592059 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-podinfo\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592199 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592085 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-credential-socket\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592199 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592128 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-socket\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592199 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592164 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-kube-api-access-bql5d\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-kube-api-access-bql5d\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592199 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592192 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-data\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592213 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-certs\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592296 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb7bc789-200b-468c-8525-5b67443e9ed9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7bc789-200b-468c-8525-5b67443e9ed9-istiod-ca-cert\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592329 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-token\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-token\") pod \"bb7bc789-200b-468c-8525-5b67443e9ed9\" (UID: \"bb7bc789-200b-468c-8525-5b67443e9ed9\") " Oct 31 02:18:52.592489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592370 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-socket" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:52.592489 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592418 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-credential-socket" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:52.592885 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592611 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-data" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:52.592885 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592688 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.592885 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592706 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.592885 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592741 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.593189 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.592995 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-certs" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:52.593359 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.593331 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb7bc789-200b-468c-8525-5b67443e9ed9-istiod-ca-cert" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:18:52.594581 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.594561 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-envoy" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:18:52.594829 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.594810 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-kube-api-access-bql5d" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "kube-api-access-bql5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:52.595234 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.595217 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-podinfo" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:18:52.595564 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.595534 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-token" pod "bb7bc789-200b-468c-8525-5b67443e9ed9" (UID: "bb7bc789-200b-468c-8525-5b67443e9ed9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:18:52.694304 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.694266 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.694304 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.694300 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bql5d\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-kube-api-access-bql5d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.694502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.694318 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.694502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.694333 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb7bc789-200b-468c-8525-5b67443e9ed9-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.694502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.694345 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:52.694502 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:52.694357 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb7bc789-200b-468c-8525-5b67443e9ed9-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:18:53.369728 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:53.369682 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" Oct 31 02:18:53.370175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:53.369747 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l" event={"ID":"bb7bc789-200b-468c-8525-5b67443e9ed9","Type":"ContainerDied","Data":"22e22230579f6cff6620e096f34c958b511d6cead05fc67f1c950ad1044d6783"} Oct 31 02:18:53.370175 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:53.369810 5090 scope.go:122] "RemoveContainer" containerID="97d8dfc88c65f951a5797f36b4b416e46828a9c1cd12b1ee85f20d58ec8e1e25" Oct 31 02:18:53.392086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:53.392058 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l"] Oct 31 02:18:53.396544 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:53.396521 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6ae519-openshift-default-54bbd75c8c-zzq7l"] Oct 31 02:18:53.745811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:53.745784 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" path="/var/lib/kubelet/pods/bb7bc789-200b-468c-8525-5b67443e9ed9/volumes" Oct 31 02:18:56.605196 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:56.605162 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:56.758925 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:56.758891 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:56.910739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:56.910631 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:57.574008 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:57.573974 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:58.077791 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:58.077755 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:58.214190 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:58.214156 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:58.391281 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:58.391188 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn"] Oct 31 02:18:58.391443 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:58.391387 5090 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" containerID="cri-o://9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79" gracePeriod=30 Oct 31 02:18:58.466151 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:58.466105 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:18:59.840377 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:18:59.840332 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b67f8_8507cc7a-393d-40c2-bd9a-8755fe6ae0bb/manager/0.log" Oct 31 02:19:00.036261 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:00.036227 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bzvnw_ec0e0324-e8d7-480a-a599-99a0ef19f3b3/manager/0.log" Oct 31 02:19:01.397406 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:01.397373 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-276kh_d7d1860c-077f-444b-bf09-7040910e43c4/manager/0.log" Oct 31 02:19:02.097695 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:02.097660 5090 patch_prober.go:29] interesting pod/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 02:19:02.097886 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:02.097741 5090 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 02:19:02.405428 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:02.405356 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:19:02.599350 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:02.599319 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:19:02.812894 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:02.812859 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:19:03.295659 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.295624 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:19:03.496232 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.496191 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:03.655410 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.655382 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 02:19:03.698926 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.698893 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-credential-socket\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699130 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.698949 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-data\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699130 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.698979 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-socket\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699130 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699018 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-envoy\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699130 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699048 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-kube-api-access-k2fj8\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-kube-api-access-k2fj8\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699130 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699080 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-istio-token\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-istio-token\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699402 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699140 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f004317f-26ec-4a67-af3a-4801711d090d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004317f-26ec-4a67-af3a-4801711d090d-istiod-ca-cert\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699402 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699187 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f004317f-26ec-4a67-af3a-4801711d090d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004317f-26ec-4a67-af3a-4801711d090d-istio-podinfo\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699402 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699271 5090 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-certs\") pod \"f004317f-26ec-4a67-af3a-4801711d090d\" (UID: \"f004317f-26ec-4a67-af3a-4801711d090d\") " Oct 31 02:19:03.699610 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699267 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-credential-socket" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:19:03.699689 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699602 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f004317f-26ec-4a67-af3a-4801711d090d-istiod-ca-cert" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 02:19:03.699778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699277 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-data" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:19:03.699778 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699513 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-certs" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:19:03.699894 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.699785 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-socket" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:19:03.701740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.701693 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-envoy" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 02:19:03.701951 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.701921 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-istio-token" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:19:03.702050 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.701992 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f004317f-26ec-4a67-af3a-4801711d090d-istio-podinfo" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 02:19:03.702123 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.702101 5090 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-kube-api-access-k2fj8" pod "f004317f-26ec-4a67-af3a-4801711d090d" (UID: "f004317f-26ec-4a67-af3a-4801711d090d"). InnerVolumeSpecName "kube-api-access-k2fj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 02:19:03.801122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801088 5090 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801122 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801121 5090 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801136 5090 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801147 5090 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801156 5090 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fj8\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-kube-api-access-k2fj8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801167 5090 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f004317f-26ec-4a67-af3a-4801711d090d-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801176 5090 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f004317f-26ec-4a67-af3a-4801711d090d-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801184 5090 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f004317f-26ec-4a67-af3a-4801711d090d-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.801324 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.801193 5090 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f004317f-26ec-4a67-af3a-4801711d090d-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 02:19:03.941511 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:03.941437 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:19:04.432202 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.432162 5090 generic.go:356] "Generic (PLEG): container finished" podID="f004317f-26ec-4a67-af3a-4801711d090d" containerID="9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79" exitCode=0 Oct 31 02:19:04.432376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.432227 5090 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" Oct 31 02:19:04.432376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.432244 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" event={"ID":"f004317f-26ec-4a67-af3a-4801711d090d","Type":"ContainerDied","Data":"9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79"} Oct 31 02:19:04.432376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.432327 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn" event={"ID":"f004317f-26ec-4a67-af3a-4801711d090d","Type":"ContainerDied","Data":"0a260a71d9f241e46b7dbea911ef8225d101e49a3881312b01f1e3a073a199f3"} Oct 31 02:19:04.432376 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.432349 5090 scope.go:122] "RemoveContainer" containerID="9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79" Oct 31 02:19:04.442011 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.441994 5090 scope.go:122] "RemoveContainer" containerID="9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79" Oct 31 02:19:04.442266 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:19:04.442249 5090 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79\": container with ID starting with 9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79 not found: ID does not exist" containerID="9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79" Oct 31 02:19:04.442309 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.442274 5090 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79"} err="failed to get container status \"9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79\": rpc error: code = NotFound desc = could not find container \"9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79\": container with ID starting with 9b2dfaf3e01440d8f6f2671898d2f58ac1db46b285daf1f022968bf22a437e79 not found: ID does not exist" Oct 31 02:19:04.450625 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.450585 5090 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn"] Oct 31 02:19:04.455934 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:04.455896 5090 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7e5955-openshift-default-56b44f8bf5-llxpn"] Oct 31 02:19:04.788693 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:19:04.788662 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:19:05.745815 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:05.745787 5090 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f004317f-26ec-4a67-af3a-4801711d090d" path="/var/lib/kubelet/pods/f004317f-26ec-4a67-af3a-4801711d090d/volumes" Oct 31 02:19:08.314638 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:08.314598 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:08.455027 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:08.454994 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:09.614209 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:09.614145 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:15.537797 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:15.537674 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:15.949498 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:15.949427 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:19:16.138340 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:16.138313 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:19:16.357559 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:16.357510 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:19:16.854911 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:16.854872 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:19:17.444345 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:17.444314 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:19:18.205828 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:18.205779 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-798bb5bfd4-v86t9_b530d2f1-8c94-43ef-8cac-5237861063bb/maas-api/0.log" Oct 31 02:19:20.813315 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:20.813272 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-68q6w_d396d6a1-dd56-41ae-b577-7010cbcc590a/postgres/0.log" Oct 31 02:19:22.567024 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:22.566989 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b67f8_8507cc7a-393d-40c2-bd9a-8755fe6ae0bb/manager/0.log" Oct 31 02:19:22.746683 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:22.746647 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bzvnw_ec0e0324-e8d7-480a-a599-99a0ef19f3b3/manager/0.log" Oct 31 02:19:24.065826 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:24.065753 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-276kh_d7d1860c-077f-444b-bf09-7040910e43c4/manager/0.log" Oct 31 02:19:24.864893 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:24.864864 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-f966s_563c1a67-983f-49b3-b52e-a57c8e515ce6/istio-proxy/0.log" Oct 31 02:19:25.600086 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:25.600052 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ddrf6_5c521031-f454-4d7a-a4d6-f460b8df6560/discovery/0.log" Oct 31 02:19:26.647262 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:26.647234 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8qmll_34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24/istio-proxy/1.log" Oct 31 02:19:26.849056 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:26.849022 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9vpv6_a3621e1c-e276-4304-a11d-8a44a339ea04/istio-proxy/0.log" Oct 31 02:19:27.431984 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:27.431912 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-ks486_9de91f6a-4968-47b5-85bb-58d455c37a07/istio-proxy/0.log" Oct 31 02:19:29.304692 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:29.304660 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54786d95-fr8g9_befed7f0-b23a-48a5-803e-6f8bf380012c/router/0.log" Oct 31 02:19:29.791795 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:29.791762 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8qmll_34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24/istio-proxy/0.log" Oct 31 02:19:34.787524 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:19:34.787489 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:19:40.542196 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:40.542163 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:42.223915 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.223887 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:42.281030 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.280996 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kprhq/must-gather-g7ssp"] Oct 31 02:19:42.281475 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281457 5090 cpu_manager.go:414] "containerMap: removing container" podUID="9a2b427a-d7ed-4b63-875e-d56ec348ccaf" containerName="maas-api" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281478 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a2b427a-d7ed-4b63-875e-d56ec348ccaf" containerName="maas-api" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281496 5090 cpu_manager.go:414] "containerMap: removing container" podUID="8a83099b-12c5-403d-8695-923fdce5db09" containerName="payload-processing" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281502 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a83099b-12c5-403d-8695-923fdce5db09" containerName="payload-processing" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281516 5090 cpu_manager.go:414] "containerMap: removing container" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281523 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281534 5090 cpu_manager.go:414] "containerMap: removing container" podUID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" containerName="payload-processing" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281540 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" containerName="payload-processing" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281554 5090 cpu_manager.go:414] "containerMap: removing container" podUID="56fa5427-486c-496c-b778-518435b1e0d3" containerName="maas-api" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281560 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56fa5427-486c-496c-b778-518435b1e0d3" containerName="maas-api" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281573 5090 cpu_manager.go:414] "containerMap: removing container" podUID="495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" containerName="payload-processing" Oct 31 02:19:42.281579 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281580 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281595 5090 cpu_manager.go:414] "containerMap: removing container" podUID="eb6eb277-79de-4620-8243-4f41396ce223" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281602 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb6eb277-79de-4620-8243-4f41396ce223" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281614 5090 cpu_manager.go:414] "containerMap: removing container" podUID="f3b2b839-b6a1-46fe-8678-c6d984684533" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281621 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b2b839-b6a1-46fe-8678-c6d984684533" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281630 5090 cpu_manager.go:414] "containerMap: removing container" podUID="60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281637 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281646 5090 cpu_manager.go:414] "containerMap: removing container" podUID="56101a31-a9b1-4b1d-9214-20d1ac327da9" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281653 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56101a31-a9b1-4b1d-9214-20d1ac327da9" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281663 5090 cpu_manager.go:414] "containerMap: removing container" podUID="6a4f1c2a-3b8d-49cc-91a0-202e959bc284" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281671 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4f1c2a-3b8d-49cc-91a0-202e959bc284" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281684 5090 cpu_manager.go:414] "containerMap: removing container" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281691 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281702 5090 cpu_manager.go:414] "containerMap: removing container" podUID="1a59d732-5ada-47c4-bc33-959413d7bd71" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281710 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a59d732-5ada-47c4-bc33-959413d7bd71" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281745 5090 cpu_manager.go:414] "containerMap: removing container" podUID="61dc5b10-f1f1-4eea-897a-b51407a7ef17" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281752 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dc5b10-f1f1-4eea-897a-b51407a7ef17" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281765 5090 cpu_manager.go:414] "containerMap: removing container" podUID="eef7beaa-db5f-408c-bd7a-b74a74248d63" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281772 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eef7beaa-db5f-408c-bd7a-b74a74248d63" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281783 5090 cpu_manager.go:414] "containerMap: removing container" podUID="8afb6d7a-d18e-400c-b16a-0a7296ae1313" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281791 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afb6d7a-d18e-400c-b16a-0a7296ae1313" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281804 5090 cpu_manager.go:414] "containerMap: removing container" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281811 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281821 5090 cpu_manager.go:414] "containerMap: removing container" podUID="b3a029da-d748-484f-a127-645719398a9c" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281828 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3a029da-d748-484f-a127-645719398a9c" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281838 5090 cpu_manager.go:414] "containerMap: removing container" podUID="3ca05e63-f923-4eec-8c2e-f52416ba67f4" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281845 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca05e63-f923-4eec-8c2e-f52416ba67f4" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281856 5090 cpu_manager.go:414] "containerMap: removing container" podUID="fc746fcd-8be9-4540-96fd-7e55187869be" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281863 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc746fcd-8be9-4540-96fd-7e55187869be" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281874 5090 cpu_manager.go:414] "containerMap: removing container" podUID="3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281880 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281892 5090 cpu_manager.go:414] "containerMap: removing container" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281900 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281910 5090 cpu_manager.go:414] "containerMap: removing container" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281917 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281934 5090 cpu_manager.go:414] "containerMap: removing container" podUID="5f19ad8a-396e-4322-abf4-f88698c9893c" containerName="main" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281942 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f19ad8a-396e-4322-abf4-f88698c9893c" containerName="main" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281963 5090 cpu_manager.go:414] "containerMap: removing container" podUID="4aa20b91-0f02-481d-9e22-b5690eb78cf8" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.281970 5090 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa20b91-0f02-481d-9e22-b5690eb78cf8" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282099 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb6eb277-79de-4620-8243-4f41396ce223" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282115 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b2b839-b6a1-46fe-8678-c6d984684533" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282123 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c7a2e5-2f4c-405d-b81d-dcb75495ccd6" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282131 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="56101a31-a9b1-4b1d-9214-20d1ac327da9" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282139 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4f1c2a-3b8d-49cc-91a0-202e959bc284" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282150 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="f004317f-26ec-4a67-af3a-4801711d090d" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282158 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a59d732-5ada-47c4-bc33-959413d7bd71" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282170 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dc5b10-f1f1-4eea-897a-b51407a7ef17" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282180 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="eef7beaa-db5f-408c-bd7a-b74a74248d63" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282189 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afb6d7a-d18e-400c-b16a-0a7296ae1313" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282199 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a52edf8-5178-419a-a452-19e019e5587a" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282210 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3a029da-d748-484f-a127-645719398a9c" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282218 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca05e63-f923-4eec-8c2e-f52416ba67f4" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282227 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc746fcd-8be9-4540-96fd-7e55187869be" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282236 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="3968f3a1-4bc1-4e90-a4b2-6d1542d6353b" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282244 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb7bc789-200b-468c-8525-5b67443e9ed9" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282252 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb06cda-3663-461c-93ca-7f63f7c12aa1" containerName="istio-proxy" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282262 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f19ad8a-396e-4322-abf4-f88698c9893c" containerName="main" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282272 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa20b91-0f02-481d-9e22-b5690eb78cf8" containerName="payload-processing" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282281 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a2b427a-d7ed-4b63-875e-d56ec348ccaf" containerName="maas-api" Oct 31 02:19:42.282254 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282291 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a83099b-12c5-403d-8695-923fdce5db09" containerName="payload-processing" Oct 31 02:19:42.284776 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282300 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17cbd44-ea63-47ff-8af6-689574aaea89" containerName="istio-proxy" Oct 31 02:19:42.284776 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282309 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf2337f-bdd6-4ed5-ad43-83c528eb8cb7" containerName="payload-processing" Oct 31 02:19:42.284776 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282319 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="56fa5427-486c-496c-b778-518435b1e0d3" containerName="maas-api" Oct 31 02:19:42.284776 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.282328 5090 memory_manager.go:365] "RemoveStaleState removing state" podUID="495a37aa-6bcb-47cc-86f6-6d6ba48dbe05" containerName="payload-processing" Oct 31 02:19:42.286152 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.286134 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.293839 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.293567 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/must-gather-g7ssp"] Oct 31 02:19:42.302687 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.302666 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kprhq\"/\"kube-root-ca.crt\"" Oct 31 02:19:42.302820 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.302701 5090 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kprhq\"/\"openshift-service-ca.crt\"" Oct 31 02:19:42.303500 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.303469 5090 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kprhq\"/\"default-dockercfg-2jdb2\"" Oct 31 02:19:42.361556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.361530 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/111057c1-2ef9-40e3-8f35-504c4648a6c8-must-gather-output\") pod \"must-gather-g7ssp\" (UID: \"111057c1-2ef9-40e3-8f35-504c4648a6c8\") " pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.361708 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.361672 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc24v\" (UniqueName: \"kubernetes.io/projected/111057c1-2ef9-40e3-8f35-504c4648a6c8-kube-api-access-mc24v\") pod \"must-gather-g7ssp\" (UID: \"111057c1-2ef9-40e3-8f35-504c4648a6c8\") " pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.369404 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.369382 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:42.463101 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.463075 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/111057c1-2ef9-40e3-8f35-504c4648a6c8-must-gather-output\") pod \"must-gather-g7ssp\" (UID: \"111057c1-2ef9-40e3-8f35-504c4648a6c8\") " pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.463265 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.463189 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc24v\" (UniqueName: \"kubernetes.io/projected/111057c1-2ef9-40e3-8f35-504c4648a6c8-kube-api-access-mc24v\") pod \"must-gather-g7ssp\" (UID: \"111057c1-2ef9-40e3-8f35-504c4648a6c8\") " pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.463428 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.463410 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/111057c1-2ef9-40e3-8f35-504c4648a6c8-must-gather-output\") pod \"must-gather-g7ssp\" (UID: \"111057c1-2ef9-40e3-8f35-504c4648a6c8\") " pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.474927 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.474873 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc24v\" (UniqueName: \"kubernetes.io/projected/111057c1-2ef9-40e3-8f35-504c4648a6c8-kube-api-access-mc24v\") pod \"must-gather-g7ssp\" (UID: \"111057c1-2ef9-40e3-8f35-504c4648a6c8\") " pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.610136 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.610106 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/must-gather-g7ssp" Oct 31 02:19:42.886871 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.886842 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/must-gather-g7ssp"] Oct 31 02:19:42.889701 ip-10-0-141-145 kubenswrapper[5090]: W1031 02:19:42.889674 5090 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod111057c1_2ef9_40e3_8f35_504c4648a6c8.slice/crio-550fc3b08a8f5f22de6b49705b4adcf515047899b9cbf30a2a85f45b9c1740bf WatchSource:0}: Error finding container 550fc3b08a8f5f22de6b49705b4adcf515047899b9cbf30a2a85f45b9c1740bf: Status 404 returned error can't find the container with id 550fc3b08a8f5f22de6b49705b4adcf515047899b9cbf30a2a85f45b9c1740bf Oct 31 02:19:42.891594 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:42.891569 5090 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 02:19:43.631790 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:43.631752 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/must-gather-g7ssp" event={"ID":"111057c1-2ef9-40e3-8f35-504c4648a6c8","Type":"ContainerStarted","Data":"550fc3b08a8f5f22de6b49705b4adcf515047899b9cbf30a2a85f45b9c1740bf"} Oct 31 02:19:44.641022 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:44.640988 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/must-gather-g7ssp" event={"ID":"111057c1-2ef9-40e3-8f35-504c4648a6c8","Type":"ContainerStarted","Data":"a7d888710696e1ec6d479d876447026d45c23fcf2d787ebdc02adadb2ca373c7"} Oct 31 02:19:44.641022 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:44.641030 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/must-gather-g7ssp" event={"ID":"111057c1-2ef9-40e3-8f35-504c4648a6c8","Type":"ContainerStarted","Data":"0de43af87952b31047657337502982879609efb5a1a67216cbf41f857279b6bd"} Oct 31 02:19:44.663098 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:44.663036 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kprhq/must-gather-g7ssp" podStartSLOduration=1.377498283 podStartE2EDuration="2.663016669s" totalImagesPullingTime="1.285518386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 02:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:19:44.662573788 +0000 UTC m=+3585.042819012" watchObservedRunningTime="2026-10-31 02:19:44.663016669 +0000 UTC m=+3585.043261896" Oct 31 02:19:46.103091 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:46.103043 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c5d98_7e587446-856d-4580-a69f-94cd5953a81d/global-pull-secret-syncer/0.log" Oct 31 02:19:46.223588 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:46.223559 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-bcgq4_57ac8842-181e-4186-b7b4-5175a511a0e6/connection-checker/0.log" Oct 31 02:19:46.251153 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:46.251130 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-d9zxv_f1b7733e-21c3-4770-bb58-d538802cbcaf/connection-checker/0.log" Oct 31 02:19:46.279217 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:46.279188 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-llbl2_22fd2129-24b4-4c50-91be-bea78425e824/connection-checker/0.log" Oct 31 02:19:46.337041 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:46.337010 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" Oct 31 02:19:46.544750 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:46.543922 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal_6555262b15c85369c5c728ca31cfdef1/haproxy/0.log" Oct 31 02:19:52.559915 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:52.556312 5090 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7rq4g_e137e152-115f-4459-80fe-a92048049da2/konnectivity-agent/0.log" limit=1024 Oct 31 02:19:53.080853 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:53.080821 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-b67f8_8507cc7a-393d-40c2-bd9a-8755fe6ae0bb/manager/0.log" Oct 31 02:19:53.110824 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:53.110734 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-bzvnw_ec0e0324-e8d7-480a-a599-99a0ef19f3b3/manager/0.log" Oct 31 02:19:53.742740 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:53.739550 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-276kh_d7d1860c-077f-444b-bf09-7040910e43c4/manager/0.log" Oct 31 02:19:54.901018 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:54.900986 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:19:54.967663 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:54.967617 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:19:55.239221 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.239141 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-8wdsg_6b85121a-782e-4f5a-b984-a775a806b231/cluster-monitoring-operator/0.log" Oct 31 02:19:55.720230 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.720182 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:19:55.769349 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.769317 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wknwn_2749cd14-51e0-4df3-b410-8357b589b1ad/node-exporter/0.log" Oct 31 02:19:55.818902 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.818872 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wknwn_2749cd14-51e0-4df3-b410-8357b589b1ad/kube-rbac-proxy/0.log" Oct 31 02:19:55.831662 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.831620 5090 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d"] Oct 31 02:19:55.838348 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.838319 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.841560 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.841534 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d"] Oct 31 02:19:55.863310 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.863283 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wknwn_2749cd14-51e0-4df3-b410-8357b589b1ad/init-textfile/0.log" Oct 31 02:19:55.894181 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.894147 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zh2\" (UniqueName: \"kubernetes.io/projected/50f96360-4b57-40f2-b7b8-d6c9fc542863-kube-api-access-s6zh2\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.894465 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.894443 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-podres\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.894615 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.894599 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-sys\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.894768 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.894749 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-lib-modules\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.894979 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.894961 5090 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-proc\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.995680 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995647 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-proc\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995741 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zh2\" (UniqueName: \"kubernetes.io/projected/50f96360-4b57-40f2-b7b8-d6c9fc542863-kube-api-access-s6zh2\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995797 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-proc\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995814 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-podres\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995861 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-sys\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995886 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-podres\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995897 5090 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-lib-modules\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.995989 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-sys\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:55.996148 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:55.996029 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50f96360-4b57-40f2-b7b8-d6c9fc542863-lib-modules\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:56.008102 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.008069 5090 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zh2\" (UniqueName: \"kubernetes.io/projected/50f96360-4b57-40f2-b7b8-d6c9fc542863-kube-api-access-s6zh2\") pod \"perf-node-gather-daemonset-wbh9d\" (UID: \"50f96360-4b57-40f2-b7b8-d6c9fc542863\") " pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:56.162556 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.162509 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:19:56.171206 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.171181 5090 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:56.304319 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.304233 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-bnsfq_c32161dd-88f4-42b2-a8dc-fc2ffdd504e4/prometheus-operator-admission-webhook/0.log" Oct 31 02:19:56.471648 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.463341 5090 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d"] Oct 31 02:19:56.630504 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.630470 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:19:56.694194 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.694164 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:19:56.771976 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.771936 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" event={"ID":"50f96360-4b57-40f2-b7b8-d6c9fc542863","Type":"ContainerStarted","Data":"df407192715f8959e75b80b3c2a5fd15b6ba2e6490985756877fec1ca145b81e"} Oct 31 02:19:56.772163 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.772013 5090 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" event={"ID":"50f96360-4b57-40f2-b7b8-d6c9fc542863","Type":"ContainerStarted","Data":"f724271276624452f1f1a46388e248d491199de389f90ff47ad0aa03a34e3de3"} Oct 31 02:19:56.773029 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.773002 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:19:56.794580 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:56.794515 5090 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" podStartSLOduration=1.79449697 podStartE2EDuration="1.79449697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 02:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 02:19:56.790782026 +0000 UTC m=+3597.171027249" watchObservedRunningTime="2026-10-31 02:19:56.79449697 +0000 UTC m=+3597.174742194" Oct 31 02:19:57.057793 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:57.055939 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:19:57.411262 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:57.411172 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:19:57.846672 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:57.845249 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:19:57.930739 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:57.929464 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-d74bn_3bf687ff-9445-49b0-804a-5eb5ca7ddedc/networking-console-plugin/0.log" Oct 31 02:19:57.968965 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:57.968936 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:19:58.035918 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:58.035884 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:19:58.713255 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:58.713224 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:19:58.727140 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:58.726245 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/2.log" Oct 31 02:19:58.750482 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:58.750452 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:19:58.825379 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:58.825346 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:19:59.185046 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:59.185015 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:19:59.311946 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:59.311914 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-79b8b67c9-w9c7m_3a1d944d-3d2c-4dc0-87d2-1c29fd90927f/console/0.log" Oct 31 02:19:59.357392 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:59.357361 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-s29ln_e6fe2a1c-6e57-426e-acc0-8f18c1e2bb8a/download-server/0.log" Oct 31 02:19:59.814958 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:59.814922 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:19:59.887487 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:19:59.887461 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:20:00.004403 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.004370 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:20:00.034053 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.034013 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-q2bv7_3e5f8170-98cf-4487-84e2-0aeccb45ba6c/volume-data-source-validator/0.log" Oct 31 02:20:00.579742 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.576301 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:20:00.626468 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.626441 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:20:00.627327 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.627300 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zkw9h_dd04b736-45e6-44cc-a032-e47ef93641a9/console-operator/1.log" Oct 31 02:20:00.649280 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.649251 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:20:00.650846 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.650823 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kjrt4_72564aed-4e32-4815-8a3e-6f61a8c36e83/ovn-acl-logging/0.log" Oct 31 02:20:00.893029 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:00.892959 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:20:01.003931 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:01.003900 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:20:01.096165 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:01.096129 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:20:01.723139 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:01.723101 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:20:01.776472 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:01.776437 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrqzl_df649d60-b1cd-408d-a0fc-e74fd86e6896/dns/0.log" Oct 31 02:20:01.809469 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:01.809386 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrqzl_df649d60-b1cd-408d-a0fc-e74fd86e6896/kube-rbac-proxy/0.log" Oct 31 02:20:01.884811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:01.884780 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mz75q_930232e3-20a2-4134-99ab-9817dede3186/dns-node-resolver/0.log" Oct 31 02:20:02.173601 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:02.173515 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:20:02.186143 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:02.186088 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:20:02.586900 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:02.584903 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-674c48bf5d-rzkcr_7779b35b-c98d-4220-9d81-1f44d9440fae/registry/0.log" Oct 31 02:20:02.641553 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:02.641519 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pmc4f_29151040-d2de-4a3c-b608-dc74c6aeee0e/node-ca/0.log" Oct 31 02:20:02.974347 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:02.974245 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:20:03.201877 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:03.201846 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-n8bs7_c88bc543-4551-48e1-9054-4ad3210fcd05/manager/0.log" Oct 31 02:20:03.268157 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:03.268122 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-8whxs_bfaf10a1-76a7-49ee-bd87-671bab789262/manager/0.log" Oct 31 02:20:03.808026 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:03.807994 5090 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kprhq/perf-node-gather-daemonset-wbh9d" Oct 31 02:20:03.831876 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:03.831841 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-f966s_563c1a67-983f-49b3-b52e-a57c8e515ce6/istio-proxy/0.log" Oct 31 02:20:03.963686 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:03.963627 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9kzpn_e2ff1e38-7f31-4da0-9e4c-6683b8553a6d/manager/0.log" Oct 31 02:20:04.087647 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:04.087545 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-f966s_563c1a67-983f-49b3-b52e-a57c8e515ce6/istio-proxy/0.log" Oct 31 02:20:04.415983 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:04.415903 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8555557b5d-s7mt2_abaa3149-5e37-43e5-8d9d-1dc6a63b86b3/manager/0.log" Oct 31 02:20:04.812110 ip-10-0-141-145 kubenswrapper[5090]: E1031 02:20:04.812063 5090 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8jv29" containerName="extractor" Oct 31 02:20:05.866635 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:05.866543 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-qlv8w_3eb1a5b5-d544-440c-8838-f7fe2a93d8a6/proxy/0.log" Oct 31 02:20:07.087452 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:07.087424 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-f966s_563c1a67-983f-49b3-b52e-a57c8e515ce6/istio-proxy/0.log" Oct 31 02:20:07.216879 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:07.216828 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ddrf6_5c521031-f454-4d7a-a4d6-f460b8df6560/discovery/0.log" Oct 31 02:20:07.417135 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:07.415808 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ddrf6_5c521031-f454-4d7a-a4d6-f460b8df6560/discovery/0.log" Oct 31 02:20:07.439863 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:07.438468 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-98f7774f-f966s_563c1a67-983f-49b3-b52e-a57c8e515ce6/istio-proxy/0.log" Oct 31 02:20:09.675142 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:09.675109 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ddrf6_5c521031-f454-4d7a-a4d6-f460b8df6560/discovery/0.log" Oct 31 02:20:09.938525 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:09.938450 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ddrf6_5c521031-f454-4d7a-a4d6-f460b8df6560/discovery/0.log" Oct 31 02:20:12.863811 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:12.863768 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8qmll_34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24/istio-proxy/0.log" Oct 31 02:20:12.867759 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:12.867737 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-8qmll_34bf4cf5-fccc-45b2-a1d1-8bc6c9e06a24/istio-proxy/1.log" Oct 31 02:20:12.886785 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:12.886766 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9vpv6_a3621e1c-e276-4304-a11d-8a44a339ea04/istio-proxy/0.log" Oct 31 02:20:12.962671 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:12.962633 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-ks486_9de91f6a-4968-47b5-85bb-58d455c37a07/istio-proxy/0.log" Oct 31 02:20:13.182959 ip-10-0-141-145 kubenswrapper[5090]: I1031 02:20:13.182876 5090 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54786d95-fr8g9_befed7f0-b23a-48a5-803e-6f8bf380012c/router/0.log"